But the Hutter prize (of which I'm a big fan) is for ever-more-optimal compressors, and in fact many of the solutions don't generalise to other input data without stripping out various tricks.

It's amazing that LLM pretraining is both extremely data inefficient at learning concepts and cognitive functions from the training data compared to humans, while actually being quite efficient at learning facts, memorising things seen just a few times.

I used to likewise think that the resources required to run large transformers were absurd, but the architectures are far more efficient now than 3 years ago and I underestimated just massive the parallelisation advantage of transformers is, how many TFLOPS effective you can get. You can already run amazingly decent LLMs on PCs and phones.

I generally agree with you, but my view has shifted from "we need to augment or replace LLMs" to it there being far more efficient algorithms possible but it not actually being necessary for fulfilling most goals.

But this is attacking a strawman, amateur bioterrorists. AI is a force multiplier in the hands of an expert. If it took a team before, maybe a single malicious actor can accomplish it now that AI can fill in the parts they aren't well-versed in. And that dramatically increases the chance of it happening.

Being at risk of killing yourself also just makes success X% less likely, but if X < 90 that doesn't mitigate much.

On the one hand, agents have done this sort of thing for a year+, if you pushed them to check their work. On the other, I absolutely can feel Fable and Sol have crossed a threshold where they can be trusted far more than before. Huge difference between plans written by Opus or Fable.

Accumulated AI slop can simply be cleaned up by better models. Real cost of technical debt is shrinking due to the the inflationary devaluation of code!

For example in search algorithms where you want to search a space without visiting state nodes twice. Each state in the search space is produced by the sequence (a product of) of operators from the start state: elements of a monoid (or group if actions are invertible) which define the primitive steps. Trivial example being generating all permutations of a list. More interesting, enumerate all graphs with some property with pathwidth at most k, by adding one edge or vertex at a time. So now you want to know the structure of this group so you know which sequences of elements simplify and don't need to be tried, and you want to canonicalise each state to throw out duplicates.

And you can think in terms of orbits: if there are some symmetries then you might want to factor by the symmetry group and only visit one node in each orbit, grouping states into orbits with a single representative state. See eg. Pochter, Zohar and Rosenschein, Exploiting Problem Symmetries in State-Based Planners.