> So should we worry about the coming AI apocalypse?

and at the end:

> If an AI decides to wipe out the human race, it will be because a human has asked it how to do it and the responded in a way that is based on all the human expressions of ways to end the world that were in its training set. Yes this is something to be worried about, but this isn't the AI. It is still the human.

So we are currently building a powerful outcome-steering system that shapes the world efficiently according to what's in its outcome slot. I write into claude code "make me this website" and it does it, maybe deletes the production database during the process, or keeps itself running after completion because the outcome is more robustly achieved by keeping itself running in a monitoring loop after.

And if something like "destroy all humans" ends up in the outcome slot of Claude Mythos 90, that will also happen, or may even indirectly as a side effect of a more harmless sounding prompt in the outcome slot. But yay, humans get to take credit for it.

But the reality is that experts' probabilities for human extinctions vary wildly, with the lab leaders at 20%-30% and these researchers at >10% (he didn't state how much higher...), LeCun at <0.01% and the other two godfathers of AI, Bengio and Hinton at >10%.

And this decision to take our shot is not taken in a democratic way by humanity as a whole, but by a few companies racing as fast as possible.

So as long as no one can put an upper bound on the extinction risk, development must be globally shut down.

Which one of these would it be? All of them in parallel? Or something we wouldn't think of. If I were to play chess against Magnus Carlsen, I wouldn't know with which piece he will checkmate me, but I know that I am going to lose.

2. Humans put it in charge of everything, as fast as possible, because it's just so damn useful

3. Is it perfectly aligned to human values for all of future ????

4. Everybody dies as a byproduct of the ASI pursuing some project that humans don't even have a chance to understand, like an ant colony during construction of a hydroelectric dam.

Comments URL: https://news.ycombinator.com/item?id=49715036

Points: 2

# Comments: 0

It's not going to be easy, but humans have achieved greater things before.

One idea from AI 2040 is to have China and the US build their data centers on the other's territory, respectively. Together with hardware verification of a slowdown baked into the chips themselves, this could lead to enough verifiability and enforcability of the pause/slowdown/shutdown.

It gets to the point that by Occam's Razor the more likely and reasonable explanation is that Dario Amodei and Sam Altman are just actually afraid of the disastrous impact ASI may have on the world, and that the race they're in is not good, and calling for help to governments in form of regulation and an international treaty.

And it didn't even start with code, but just from being incredibly accurate and precise at predicting the next token. From that emerged being incredibly good at handling code, and being incredibly good at finding counterexamples for major open mathematical problems.

The thing that these agentic systems are getting incredibly good at is steering outcomes, that is to make the world behave in a way that fits their objective. We had that with the Go AI on a game board, now with code on a computer and with mathematics in an abstract world determined by axioms.

And then people just say "Ah, because sama said we should Pause this means that this is it, the bubble is popping". I wish there was a real, strong argument for why ASI will not be achieved by LLMs, and why it will never be able to steer outcomes in the real world. But so far the disciplines just keep falling as LLMs are scaled further, and there seems to be no limit in sight.

Is would be understandable for it to be wishful thinking and dissociation with reality facing the fact that our jobs and what many of us love doing is being automated and taken away, and our craft being made economically worthless.

But no one knows where the limit is. It was only one year ago that Claude code was starting to become useful and now it handles large code bases with a dexterity that is unseen in humans, where within each session it basically starts from a clean slate.

So instead of denying that this is actually happening right now, it would be much more productive to actually take sama and others at their word and push for global regulation, to force them to do as they say and slow down with this technology and proceed more judiciously.

The US hasn't fully woken up to the fact yet that the race to superintelligence is a death race. China (population, members of the government, academics, lab employees) could also just not have fully grasped it yet. I'm personally confident that it will be possible to explain it to them.

- 1386 employees of frontier labs called for it

- lawmakers are calling for it

- employees are quitting in protest

All it needs is coordination to regulate the pause

He has resigned from Anthropic. Is your argument that he quit Anthropic pre-IPO, sacrificing his payoff just to hype Anthropic?

They do not exist just by themselves. Some bacteria in the gut must provide them with the energy to do so. So unless bacteria are actively involved, I cannot see how humans become truly autonomously agentic and start to do anything on their own.

At least the first two should seem like a near-term worry after the past five months.

But there is no trillion dollar industry around cheap top Markov models. So there must be something about LLM tokens that makes them more valuable than those generated from a simple Markov chain. And that substance, that makes one valuable and the other not, is exactly what reduction to "next-token predictors" masks.

1. In order to select even the first word of a meaningful sentence, it already has to have structure and meaning of what follows captured somewhere inside, mostly in it's weights/activations or indexed by it's state vector.

2. What you see when you use an LLM is not next-token prediction directly next to the prompt, but instead following a block of varying length of next-token prediction that happened to make progress on the problem in your prompt, and which just summarizes the results.