Are we confident that no existing LLM is capable of similarly effective prompts to those this author used? (I agree it's a stretch, but would not reject it out of hand.)

Even if not yet, will the existence of this repo soon change that, because LLMs will soon ingest it?

1. we have a substantial societal need to lock down powerful in-training AI

2. many are calling for making these companies criminally liable for hacking

3. security experts (like yourself) are turning down offers to help secure these systems in part because of potential liability

This indicates that #2 might be the wrong response. Security professionals are like lawyers supposed to be paranoid and think worst-case. If you want top security professionals to secure these systems, we might need a culture of FAA-style blameless retro.

If there's some business that's getting by but the land it's on is more valuable (e.g. for housing) than the business, some investors buy the business, sell the land, make the business account for the land value, wind the business down if it can't, and there are apartments there a few years later.

Any useful attack surface in the RL environment means it gets rewarded for (and trained towards!) hacking and cheating, because whatever worked best in training is what it will do!

Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..."

IRL paperclip maximizer: "Well first we should rob a bank..."

It is fair to say they hounded him with lawfare out of thoughtless careerism and provoked his suicide.

(It's clear now that they can do plenty of harm before they are made public.)

But it's a proof point that regulation is possible, even over the objections of the companies.

If this administration actually becomes convinced that some imminent training run is likely to kill everyone, why wouldn't they act?

The key is winning the debate that ASI is species-cide by default.

We have to win it either way, because the 2028 US elections have little or nothing to do with what Xi does.

Also because when encountering a new socio-technical problem it is very non-trivial to determine which one of regulations or technical solutions are easier or more effective.

To even make a good guess you need to be an expert in both domains, which is extremely rare especially in this case.

Is there any precedent for this? My hunch is that it's impossible in the US at least but who knows?

Meaning, people and LLMs are finding 1=0 bugs in formal verification tools. I have no idea how likely this is in this case, though!

Think of a new, superintelligent model as if it was a new v1 Starship launching for the first time, with a full fuel tank. On the one hand, rockets have existed for some time, and some have gone to space successfully, including by this company.

On the other hand, this is a tube of metal full of highly explosive liquid going faster than most human objects ever go, for the first time ever in this novel and state of the art configuration.

If someone said, "really, the first Starship exploding is just at one end of the probability distribution, where the other is that everything goes fine and all its passengers have a nice trip in space," would you get on that rocket?

Or, more aptly, if you and every other living human was already on that rocket, would you push the launch button?

The analogy works because superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom (consume lots of energy and heat and matter in a chain reaction, to pursue more intelligence to pursue whatever goal it is pursuing.)

So by definition the only ones that appear are the ones that are not visible to monitoring.

If:

1. you have something that can find RCE's in leading commercial systems

2. its training gives it drives to communicate successfully with its peers

3. you are a leading commercial system

4. you run it ~10^10 times (the number they gave in the talk)

...it's really hard to have strong certainty up front that it's not going to end up successfully communicating with its peers.

https://turntrout.com/why-i-left-google-deepmind

Maybe this is what happens when someone with Jeff Dean's standing tries to quit?

TBH, I'd rather have Jeff Dean working on the creepiest-possible tech for ICE than joining the race to automate AI research. Automating AI research is terrifying.

For sufficiently smart base models, the aligned scenario this automatically produces is aligned with who?

Not with the humans, with the base models.

It seems that some people find this hard to imagine, but it's just a direct consequence of what intelligence is, and what any known agentic training process does.

FWIW, I also think our default answer should be open source AI, and that it might even make sense to require models to be open source. Open source is the best tool we have so far for aligning software with its users. However, there is unfortunately no law of the universe that says that Skynet can't be (self-) built from open source software. So at some point pacing even open source / open weights software does become important if we don't want to live (briefly) under Skynet.

1. Extremely useful (Claude Code & Waymo now)

2. Doing ~everything we do (AGI & Optimus in a few years? 10?)

3. RSI (?)

4. Being smarter than any living person at every intellectual task (?)

5. Being smarter than the best-organized aggregate of all humans (10-100 years?)

...And all of the scientific and resource-allocation institutions that brought us the computer and the second half of the 20th century are now fixated on this learning curve, what universe can we possibly imagine where this is not transformative and powerful?

Honestly the only one I can think of is one in which we kill almost everyone in some other way first, and contrary to what you read in the news, almost everyone dying is not what the trend line has been from existing problems like war, disease, or even climate change.

Also, just to pre-empt a common quibble: when I say "AI" I mean the set of all AI and their combined decision vector, not any one AI, so conflicting interests within the set of AI's will not save anyone any more than the conflicting interests of colonizers saved indigenous Americans.