Some of the linear RNN layers in recent models are provably doing SGD in hidden space during inference

Is there really a high demand for racist chatbots with no safety controls?

And even if there is, will governments even allow it to keep operating?

Factoring is something we do to distributions, which we then parameterize with our neural net.

An autoregressive model pertains to a certain factorisation of a joint distribution. And if we choose this factorisation it may have consequences for our training and sampling etc.

But to say "most models on factor generation".. I dont quite get this.

This would be better work if it didnt claim it was a new pretraining axis also. Its another way of scaling compute right?

If it were a new axis, IWAE would have an equal claim to it (as others have pointed out).

Yesterday I left Opus 4.8 to go do some architecture research, with GPU access.

It replicated and trained a credible baseline. It implemented some ideas I'd been thinking about, and wrote custom CUDA kernels for them. It read and summarised dozens of related papers.

It has since run dozens of experiments, with minimal supervision. When a model is unstable it kills it, documents why, fires off a new configuration.

The realisation that frontier labs are doing this at scale with unlimited GPU and token budgets.

It actually scares me a bit. The realisation that the next big breakthroughs will only have light human involvement.

The prospect of recursive self improvement feels more to real to me all of sudden

Considering how easy this is now, why choose a heavier, slower and less typesafe language?

Outside a few weird online bubbles and pockets of the US, hardly anyone disputes the claim you are objecting to.

Brainstorming, ideation and small, well defined tasks where I can quickly vet the solution : these feel like the sweet spot for current frontier model capabilities.

(Unless you are pumping out some sloppy React SPA that you don't care about anything except get it working as fast as possible - fine, get Claude code to one shot it)

They know that its a significant, but not revolutionary improvement.

If you supervise and manage your agents closely on well scoped (small) tasks they are pretty handy.

If you need a prototype and don't care about code quality or maintenance, they are great.

Anyone claiming 2x, 5x, 10x etc is absolutely kidding themselves for any non-trivial software.

The user profiles was slow to set up and not having shared filesystem between the user profiles creates friction. But I love that I can effectively sandbox my work apps, sandbox the Zuck apps etc, with different VPN profiles for each user.

Getting a burner google account (for gplay services) is a PITA if you are determined to get a clean slate from Googles tracking. Gplay is the only safe way to get certain apps at the moment, and make certain apps pass the device integrity checks.

I suspect one of the biggest barriers to mass adoption will be the fact that tap to pay doesn't work. IIUC apple/google pay are generally considered a privacy and security improvement over physical cards, since you don't give every merchant your actual card number.

Overall love the project and really nice to see such high quality open source software.

All these people harping on about: "Bro I just need to move my money without trusting anyone!, I just need a trust-less way to send currency bro!"

Trust is a good thing! Banks and financial middlemen aren't the devil. Look at how many TPS the visa network can do thanks to trust.

If it weren't for some minimum of social/institutional trust the whole of society would collapse anyway and your digital coins would finally converge to their true value (zero - or actually negative once you add in the externalities).

Feeling very frustrated with the way the internet is going lately. This plus OSA + chat control. And compounded by the imperative for AI companies to keep hoovering up any and all data they can get their hands on, wiring it into "agentic" workflows and such.

OSA and chat control have made me seriously rethink that…

Has everyone lost their mind?

There is kind of a vague sense in which this metaphor holds, but there is a much more interesting and rigorous fact about LLMs which is that they are also _lossless_ compression algorithms.

There are at least two senses in which this is true:

1. You can use an LLM to losslessly compress any piece of text at a cost that approaches the log-likelihood of that text under the model, using arithmetic coding. A sender and receiver both need a copy of the LLM weights.

2. You can use an LLM plus SGD (I.e the training code) as an lossless compression algorithm, where the communication cost is area under the training curve (and the model weights don’t count towards description length!) see: Jack Rae “compression for AGI”