I'm tired of this cautious optimist talk. It is inevitable that without regulation, AI will eclipse humans in every possible domain, leaving everyone without direct control over the most powerful models completely powerless, economically redundant, and at the mercy of the new AI hegemony. Anything which doesn't confront this possibility honestly is just lying to people.

I don't think this will happen in this universe though, as we will get a fair amount of "warning shots" on what AI can do on its own and how those goals necessarily disempower, threaten, and endanger human welfare. This will lead to a significant regulatory regime, retraction of the capitalist mode of AI buildout to a more state-level one, and perhaps a soft dictatorship under a multi-polar paced frontier.

The average person's life will improve though as AI will enable many breakthroughs in medicine, technology, etc which won't require superintelligence. However we will live under the perpetual shadow of the titan we put on a leash and managed to pilfer a pittance of the singularity's bounty from.

It’s similar to the No True Scotsman fallacy: any bad outcome due to AI could be dismissed as “well no aligned AI would do that”. Alignment becomes an asymptotic north star which very well could be theoretically impossible, but would be impossible to know whether it was.

The most common issue I have with them is their insistence on rigid classification: "AI is a tool, it doesn't have intentions, or desires, only humans do", as if that rhetorical flourish eliminates years of alignment and capabilities research. There is a desire to characterize all the developments in LLM's as some sort of parlor trick which by definition cannot be greater than the sum of its parts (i.e. the outputs of an LLM can never be greater than the sum of its training data, genuine artificial "intelligence" is impossible, etc)

Also those at Anthropic genuinely believe AI could lead to doom, that's why they started the company, they believe only they can be stewards of digital gods. Whether you believe it or not, they genuinely do, and wouldn't voluntarily sow fear, uncertainty and doubt in their product when it doesn't guarantee benefits.

This is an instance of work in a genre I call "AI lab grievance poetry" where a vast collection of criticisms are levied against AI labs, with specifc ad-hominems towards the executives, while avoiding the core of their arguments about AI risk

Also an initial, cumbersome, complex proof is the first step to a more understandable, formalized proof. AI created a formal proof of Fermat's last theorem. I'm sure the initial proof was not comprehensible to all but a small subset of mathematicians anyway.

People are grasping at straws it seems to dismiss the power of this new model they may have. Hate OpenAI for any reason you want, but denying the capabilities of models has been a losing game for the past 5 years.