Or, another way to put it: an extremely powerful military technology the careless usage of which could destroy the human civilization was only used in one war in history and killed less than a million people, and then it was never used again over the next 80 years. I think this is a success story for humanity, and the NPT was an amazing feat of coordination.

> And like, now look at the status quo of nuclear treaties and agreements and proliferation.

I mean, sure, but giving every state (much less every person) the ability to make nuclear weaponry wouldn't make things any better.

I think the reason we think of this differently is because you believe that states are by default... corrupt, maybe, or unreliable, or evil, while individuals are mostly fine. Whereas I think that most people can't be trusted to make correct decisions on topics like "should we use this dangerous technology", while states at least have some decent track record at this.

A base model is absolutely not aligned. Pretraining doesn't teach an LLM to mimic the average human - doing so would make an LLM perform quite badly at predicting most of the dataset. It teaches it to mimic all possible humans¹, inferring what sort of persona to take depending on the context. A base model can indeed convincingly act like an "average person in the developed world", including by making the same sort of moral decisions that such a person would do... but it can also convincingly act like a shitty human, or like Hitler², or like any other actor that left its traces in the training dataset. A pretrained LLM therefore contains multitudes of personas, some of which would be considered aligned if you could make the LLM elicit them robustly, and most of them wouldn't be. But then you're left with the problem of how to make a base model elicit a very specific persona robustly even in out-of-distribution scenarios, which is not necessarily easier than solving alignment any other way.

(Another note is that the question of how aligned base models are is rather academic because almost nobody uses them anyway, because it's hard to get powerful capabilities by pretraining alone. Nowadays most of the frontier models' programming and math abilities are driven by RLVR.)

¹ Really "all generators of text that went into the training dataset".

² Even after RL training LLMs still retain those personals and can be convinced to elicit them quite easily, though it takes a tiny bit of finetuning: see https://arxiv.org/pdf/2512.09742.

I'd trust the latter and I think it's an easy choice. I'm sure it feels bad to not be in the group with access to the scary weapons, but do remember that out of your two groups, the "collective humanity" one is the one with the literal terrorists, which do in fact exist and aren't a myth. For comparison, observe how the concentration of ability to produce nuclear weaponry in the hands of only a few states did, historically, work to prevent both a nuclear war and any nuclear terrorism.

> And the rest of us have no way to stop it or defend.

If you did have access, what'd be your defense plan? Biorisk is one of the most attacker-favoring fields imaginable, so a world where there's an equilibrium between attackers and defenders in bioweaponry research (the same way cybersecurity currently works) would be quite terrible. Your best bet would probably be to take a new vaccine each time a new engineered disease comes out, and hope that you're never one of the suckers who got infected before the vaccine was developed, and that the accumulated side effects from multiple experimental vaccines don't kill you too quickly.

https://github.com/redlib-org/redlib

  A) ASI is developed and massively overshadows the rest of the world economy 
  B) the world still has rule of law, contracts, business, well-developed finance, etc

which seems to have eventually been merged into vllm: https://docs.vllm.ai/en/latest/features/batch_invariance/

So you can get determinism locally. On a cursory search I wasn't able to find any LLM provider advertising determism; if you need it for research you might have to rent a dedicated GPU pod and run vllm there with the appropriate settings.

Comments URL: https://news.ycombinator.com/item?id=48316686

Points: 1

# Comments: 0

In fact, the repo says:

  This is my social commentary on the "Permanent Upper Class" philosophical virus that has spread through San Francisco tech communities like the plague in the last year or so. You can read the original blog post [here].
  While the game tells a grim story, my goal with this is to show how ridiculous this way of thinking is. I'm actually more optimistic that humans will find a way to prevent this future but if we don't then I suppose none of us will don top hats for eternity.