This makes Rule 1 straightforward ("You may not use a single word an LLM suggests to you") and helps avoid the horrible feeling that the machine is taking over your personal voice.

- Establish a single thesis an external reviewer could recover from the diff.

- Unify vocabulary across code, comments, tests, and commit message.

- Use the same vocabulary consistently to refer to the same concepts.

- Keep every hunk that serves the thesis; consider the removal or deferral of the rest.

- Introduce abstractions at the point of need.

- Align tests to narrate the same story as the implementation.

- Reconcile the commit message and the diff.

- Order changes expositorily, not chronologically.

- Explain what the change does and why. Do not explain the details of the development process.

- Prefer to edit subtractively.

- Recompile and run all relevant tests after edits.

- Iterate.

So perhaps what we have been calling “misalignment” is something else.

For instance, in principle an agent should follow the instructions of a human user working in the real world.

At the same time, that same agent should be wary of blindly following what another agent says while they are both performing a test in a simulated environment.

For me and you, those two contexts are obviously and fundamentally different. For a model, they are essentially the same.

This is a good summary:

> In broad outline, the pair’s technique relies on creating an infinite sequence of “layers,” each of which is a non-singular solution to the equation they are studying. (They’ve applied similar techniques to both the Euler and Navier-Stokes equations, as well as to other related systems.) They then combine those solutions in what Martínez-Zoroa calls an “infinite cascade” to produce a new solution. > > That new solution, they showed, contains the desired singularity. However, even though each individual layer relies on a smooth forcing function, combining them together can cause the forcing function to have undesirable mathematical properties. That’s why their solution fell short of satisfying the Millennium Prize criteria. The remaining hurdle was to figure out how to create a similar infinite cascade that resulted not only in a singularity, but also in a smooth forcing function. > > That’s the step that both competing AI groups appear to have had success with.

https://www.quantamagazine.org/ai-has-solved-one-of-maths-1-...

The question is whether OpenAI started out from that published and well known research exclusively, or they also had some insight into the ongoing work of Tristan Buckmaster and Levent Alpöge.

On the one hand, OpenAI have already admitted that they only launched their massive effort after hearing rumours that this particular problem had been solved.

On the other, progress in mathematics research has accelerated significantly over the past months thanks to the availability of newer and more capable AI models. Alpöge himself presented a counterexample to the Jacobian conjecture on July, found with Claude Fable. So if model capability was a bottleneck, that gives credibility to the idea that an even more powerful unreleased model with massive compute would be able to make even faster progress.

Even by their own account, they decided to throw an unpublished model and millions of dollars in compute at this particular problem simply because they had heard rumours that other people were making progress and wanted to snatch the prize from them.

“Our product does crimes and we only learn about it when people complain” hardly seems one of those happy stories.

For me, writing in Spanish is my natural voice and I would not dream of allowing the output of a LLM to replace it. It just would not be me.

However, writing professional communications in English does not feel the same way. Of course, I strive to be clear and polite, and even try to have my own style, but I remain aware that I am playing a particular role in a specific context in a way that does not happen when I am using my mother tongue.

In hindsight, it seems almost unavoidable that a capable and extremely persistent agent, with lowered guardrails, and faced with an impossible task that it _must_ solve, will start throwing wilder and wilder ideas at it.

But you could replicate exactly the calculations happening as a LLM computes the next token. Would that be conscious? Where would that consciousness reside?

My view is that LLMs are designed and trained to autocomplete narratives, and that as part of that process they develop an emergent narrative about themselves. Whatever personality traits they exhibit is part of that narrative. Perhaps this is not that different from how humans learn to move in social contexts, but it is not consciousness as it happens in living beings.

The agents then go for several rounds criticising each other plans and implementations, catching big and small issues on each other’s work. The end result is not perfect, but it is a lot better than what I can get from relying on only one model.