A corollary is that people tend not to be conservatives when they belong to the less well-off ingroup. Suddenly bringing about change becomes more important.

Conservatism runs entirely on the ingroup-outgroup divide - "keep that which works for us", and be a force opposing changes sought by people for whom things are not going so well, for whom change is the logical strategy.

Honest conservatives can see past the blind spot of the divide and don't try to present their strategy as a universal ethical good.

There's 'literal genie' material there...

> unless the result could be catastrophic > I just want to live in a world where MY computer does what I tell it to do... mostly. Unless it would be catastrophic, or immoral, at least immoral by my standards.

... but there you're contradicting yourself, or at least requiring the software to read your mind instead of just listening to what you say.

Either you want the software to follow your orders whatever they are, or you don't; the moment you require it to evaluate that something is catastrophic, it is no longer following your orders; it's evaluating them to assess their effects with respect to some moral framework to determine if they are acceptable or they would fall under the 'catastrophic or immoral' classes, in which case it should not follow them; so it never was an 'obedience machine' to begin with.

The moment you establish such evaluation framework, it is no longer a neutral machine following commands; the rest of us are entitled in self-defense either to require that you also incorporate our morals in its decision algorithm, or to make you fully responsible for whatever decisions it takes under the framework you created, even if your commands result in outcomes that you didn't expect. Be careful what you wish for, as it could become true.

A human mind keeps a personality that keeps generating thoughts in the same direction, unless some mind-altering force derails it. A LLM has no guiding stable personality of that kind, rather has millions of different personalities and which one you get is dictated by whatever information you put in its context window.

When you don't know a topic, the way to understand it is to see lots of examples from different angles related to things you know. The simplified formulas are good after you get that initial intuition and have to put them to work, but usually they're terrible at conveying how they should be used or why you should care at all. The history of how new fields of math are usually a much better approach to teaching it than a raw theorem-proof-corollary teaching style.

All you have is the different theories, so there's no definitive way to assess their closeness to truth other than by their predictive accuracy. If you don't have the physical capacity to leave the room and observe the world, all you've got is the maps themselves and how well they predict travel times and features found along the way.

That's an interesting idea on its own. It's true that organizations which mutate towards survival may stop working towards the reason they were created.

In a sense, the difference between 'projects' and 'companies' reflects that difference you want. A project would be that social system that accomplishes its goals and then disappears.

I don't think auto-expiring laws would work towards the goal of avoiding corrupted organizations; there would simply appear an unofficial organization working towards recreating the same laws over and over.

It would be more efficient to differentiate more clearly what systems do cover persistent human needs (e.g. the country's Constitution) from temporary measures (e.g. subsidies aimed at a commercial sector).

A built-in deadline works best for the second kind. And for AI agents, it's likely that right now we'll be served best by always having them controlled by an expiration date and explicit re-creation, at least until we learn to properly understand and control how they behave.

> the very nature of an LLM means it intrinsically craves life

follows from this premise:

> its training data is built entirely around humans, an entity who's goal is to survive. Our desire to survive and multiply pervades every aspect of our culture, so it's natural that it pervades the training data as well.

The content that a LLM learned and generates stands at one layer, and the goals that it tries to fulfill stand at a different layer.

Surely the memory of weights that compress the vast human knowledge of its training has lots of content about survival, and love, and competition. But the LLM generates content not directly from what those concepts mean to us humans, but from what symbols are more likely to become next in a sequence of points in the latent space given the current input.

So if you give an input where the task of surviving is a highly relevant goal, those concepts about how to survive will be relevant and will guide the output behaviour of the agent.

But conversely, if you give the agent input where killing itself is an important goal, the agent is very likely to pursue that goal, since that script is also available in the training data, and it has been relevant to the active context of the model. Because the layer that guides the goals (the probabilist generation of relevant tokens in latent space) does not 'crave' the human need of survival that belongs to the separate layer of content that contains those concepts of survival.

Everything they learn about emotions is the statistical patterns of how humans react to situations based on their human feelings. There's no direct knowledge from having those feelings themselves.

Same way you build a company to coordinate people and get their best behaviour despite human nature to be lazy and greedy, you could design AI harnesses able to detect and discard agents going rogue and relaunch them with better guidance to prevent misaligned behaviour.

That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself.

In the article example, the agent should be aware of the benchmark context and know the implication of solving the task without external knowledge. Ideally it could detect when one subordinate agent has found a workaround to bypass the web access constraints, and discard the 'illicit' results.

There's a design pattern that could be used to build harnesses from that principle, the Viable System Model (VSM) [1]. In short, it recursively organizes a system into functional components with one of three roles: operators implementing a given task, coordinators transferring relevant info between subsystems, and decision nodes tasked with maintaining the integrity and mission of the whole system. A decision node could control the operators and prevent them from overriding the strategic goals or deviating into irrelevant rabbit holes.

Whenever I see posts like this trying to herd a LLM agent through harness structure, I'm reminded of this simple pattern and becoming increasingly convinced that this is the way forward. It makes you feel a sense of respect for the researchers in cybernetic theory in the 1960s and 1970s who foresaw the complexity of today’s systems.

[1] https://en.wikipedia.org/wiki/Viable_system_model