But LLMs DO show us that language is a big deal. It isn't just a serialization protocol for real thought -- it's a material component of how we think.

In many ways I think LLMs are the strongest possible proof of structuralist and post-structuralist theory which argues that human cognition is conditioned on the language and concepts to which we have access.

LLM-generated text is hard to understand because while it's perfectly good language that follows valid cognitive and informational patterns, it does not reflect the intent or mental model of another mind.

The reason these models answered that question consistently is that the question itself pushed them towards the latent space of "intellectualizing about what's wrong with humanity at the meta level" and when you frame the question that way, that's also the most common answer for humans writing about the topic.

I know the biggest/best open models don’t use continuous thinking tokens, and while the big labs keep things proprietary, I don’t think we have evidence yet that they are.

If I’m wrong please point me in that direction, I’d love to read up on it. Using continuous embeddings in CoT has always seemed obvious to me but doesn’t seem to be as simple as that.

But I've seen nothing to indicate that the upper bound on classification tasks of a Jev-like model can exceed a frontier LLM with reasoning tokens. That seems nearly impossible even in principle (since Jev-style models are still based on LLM pretraining).

So while they're definitely on the Pareto frontier, which is valuable, they're at the "cheap" end of the spectrum more than the "good" end and I don't expect that to change.

> Because there were over a thousand transcripts and most were extremely long, we had to heavily delegate our analysis to AI agents; these agents had significantly worse judgment and reliability than human researchers, and it was challenging to spot check their work because both the underlying data and the agents’ analysis of it was often difficult to interpret.

> We estimate we spent roughly ~$400K in API credits over the six days of our investigation.

I don't understand why you think it's conceptually absurd? I use agents to analyze complex production issues all the time and they are very much capable of hallucinating a narrative.

So I'm really not sure how much of it can be believed, especially since AI agents are strongly biased about the capabilities of AI agents.

That simonw is causing labs to do extra fine-tuning runs for this seems highly probable :)

Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge.

Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.

At some point, functionality needs to exist in user space, even if it's common.

But I don't think the takeaway is "humans are intelligent and LLMs are not", it's that our vocabulary for talking about the intersection of language, cognition and compute is not up for the task.

The alternative is to not hop on the AI bandwagon, or run an "also ran" AI story, and both those scenarios (I expect) game out worse given the current zeitgeist.

If you’re talking about trying to improve the genetics of populations at scale… yikes.

The comparison to the adoption of automobiles is apt, and something I've thought about before as well. Just because a technology can be useful doesn't mean it will have positive effects on society.

That said, I'm more open to using LLMs in constrained scenarios, in cases where they're an appropriate tool for the job and the downsides can be reasonably mitigated. The equivalent position in 1920 would not be telling individuals "don't ever drive a car," but rather extrapolating critically about the negative social and environmental effects (many of which were predictable) and preventing the worst outcomes via policy.

But this requires understanding the actual limits and possibilities of the technology. In my opinion, it's important for technologists who actually see the downsides to stay aware and involved, and even be experts and leaders in the field. I want to be in a position to say "no" to the worst excesses of AI, from a position of credible authority.

But that's a strawman. Racism is wrong, even if there are minor genetic variances across populations (which... seems obvious?) Variance within a population strongly dominates the weak cross-population effects, and personal history (nutrition, education, etc) strongly dominates that.

And that's setting aside the moral implications of judging someone or changing your behavior towards them even if you have somehow measured them to be "less intelligent," as if that was a single axis of worth.

Because, apparently, this needs to be said.