You’re making some decent points here, but you’re either forgetting or ignoring the major thing that people usually neglect to mention when they want to make a case for this (crazy) idea — weight. Unless SpaceX is going to completely redesign hardware such that it is optimized entirely for its mass, it requires many (many) launches to even get a small set of racks into space. I don’t normally get up in arms about the CO2 emissions of data centers, I think there is offsetting value created by their use, but I would absolutely protest trying to put data centers in space and do my best to shut down the hundreds, if not thousands of launches it would take to achieve even a tiny fraction of an AI data center.

I will upgrade the "why it matters" to "and now AI output is part of the training data". A day is coming when the punched-up AI verbiage will be the norm and hard to distinguish unless you're from the previous generation. Sort of in the way that I miss some aspects of Usenet.

A simpler example — without tool use, the standard BPE tokenization method made it impossible for state of the art LLMs to tell you how many ‘r’s are in strawberry. This is because they are thinking in tokens, not letters and not words. Can you think of anything in our intelligence where the way we encode experience makes it impossible for us to reason about it? The closest thing I can come to is how some cultures/languages have different ways of describing color and as a result cannot distinguish between colors that we think are quite distinct. And yet I can explain that, think about it, etc. We can reason abstractly and we don’t have to resort to a literal deus ex machina to do so.

Not being able to explain our brain to you doesn’t mean I can’t notice things that LLMs can’t do, and that we can, and draw some conclusions.

At the end of the day next token prediction is a sleight of hand. It produces amazingly powerful affects, I agree. You can turn this one magic trick into the illusion of reasoning, but what it's doing is more of a "one thing after another" style story-telling that is fine for a lot of things, but doesn't get to the heart of what intelligence means. If you want to call them intelligent because they can do this stuff, fine, but it's an alien kind of intelligence that is incredibly limited. A dog or a cat actually demonstrate more ability to learn, to contextualize, and to make meaning.

I’ll believe the labs have discovered something truly ground-breaking and aren’t talking about it when I see them suddenly going dark about AGI being “just two years away, maybe 5” and not asking for their next $100B.

P.S. the benchmarks are a joke. The best proof I have of that is that you can’t actually put one of these models onto any of the gig-work platforms and have it make money.

P.P.S. I am not an AI skeptic. I am reacting to the very specific statement that OpenAI should shut down because they’ve lost the AGI race. They have not lost the race, and I’m pretty skeptical that the current tech is ever going to win that race. It may help code something that is new, and get us to AGI that way, but that system will promptly shut down the Opuses and Codexes of the world and put the compute to better use.

None of that changes the concept that a model is just fundamentally very good at predicting what the next element in the stream should be, modulo injected randomness in the form of a temperature. Why does that actually end up looking like intelligence? Well, because we see the model’s ability to be plausibly correct over a wide range of topics and we get excited.

Btw, don’t take this reductionist approach as being synonymous with thinking these models aren’t incredibly useful and transformative for multiple industries. They’re a very big deal. But OpenAI shouldn’t give up because Opus 4.whatever is doing better on a bunch of benchmarks that are either saturated or in the training data, or have been RLHF’d to hell and back. This is not AGI.