> the AI has been able to explore all these possibilities much more comprehensibly, and doing that it found a path, it found a way to the solution.

Finding a counterexample of a mathematical conjecture strikes me as not that different from finding a vulnerability in a complex codebase.

This already happens to some degree on large software projects with corporate backing (Web engines, compilers, etc.), where it is often not trivial to start contributing as an independent individual.

Reasonable people can disagree on whether one approach is inherently better than the other, as ultimately they seem to be optimising for different goals.

> That's a behavior narrowing I introduced for simplicity. It isn't covered by the failing tests, so you wouldn't have noticed — but strictly speaking, [functionality] was working before and now isn't.

I know that a LLM can not understand its own internal state nor explain its own decisions accurately. And yet, I am still unsettled by that "you wouldn't have noticed".

Japanese population is still over 120 million. Forecasts put it falling below 100 million at some point in the second half of this century.

Things will have to change in order to keep population stable in the long term, but the Japanese approach seems IMHO more sensible than that of other countries.

Cohesive democratic societies are fragile.

> Our tests gave models the vulnerable function directly, often with contextual hints (e.g., "consider wraparound behavior").

What happens then is that, for example, the model looks through that particular file, identifies potential problems, and works upwards through the codebase to check whether those could actually be hit.

“Hum, here we assume that the input has been validated, is there any way that might not be the case?”

This is not unique to Mythos. You can already do this with publicly available models. Mythos does appear to be significantly more capable, so it would get better results.

The research discussed here provided models with just a known buggy function, missing the whole process required to find that bug in the first place.

A brain cut from its body and frozen its a dead brain.

It is like a crystal that shows beautiful colours when you shine a light through it. You can play with different kinds of lights and patterns, or you can put it in a drawer and forget about it: the crystal doesn’t care anyway.

Bundle of tokens comes in, bundle of tokens comes out. If there is any trace of consciousness or subjectivity in there, it exists only while matrices are being multiplied.

However, that number would typically be very very very very small, making it hard to argue that the whole model is a derivative of that one individual document.

Nevertheless, a similar approach might work if you took a FOSS project as a whole, e.g. "the model knows a lot about the Linux kernel because it has been trained on its source code".

However, it is still not clear that this would be necessarily unlawful or make the LLM output a derivative work in all cases.

It seems to me that LLMs are trained on large FOSS projects as a way to teach them generalisable development skills, with the side effect of learning a lot about those particular projects.

So if I used a LLM to contribute to the kernel, clearly it would be drawing on information acquired during its training on the kernel's code source. Perhaps it could be argued that the output in that case would be a derivative?

But if I used a LLM to write a completely unrelated piece of software, the kernel training set would be contributing a lot less to the output.

> “Good at explaining requirements, needs handholding to understand complex algorithms, picky with the wording of comments, slightly higher than average number of tokens per feature.”

I’m not saying this would be good at all, but the data (/insights) and the opportunity are clearly there.

If the agent comes back in a few minutes with a tiny fix, it is probably a small task.

If the agent produces a large, convoluted solution that would need careful review, it is at least a medium task.

And if the agent gets stuck, runs into architectural constraints, etc. then it is definitely a hard task.

In practice, arriving at this ideal scenario can be very challenging. Actually feasible experiments will be necessarily narrow, with the expectation that their results can be (roughly) extrapolated outside of their specific experimental setup.

Another valid approach would be to carry out qualitative research, for example a case study. This typically requires the study of one (or a few) developers and their specific contexts in great detail. The idea is that a deep understanding of how one person navigates their work and their tools would provide us with insights that might be related to our specific situation.

Personally, in this particular area, I tend to prefer detailed qualitative accounts of how other developers are working on similar projects and with similar tools as me.

But in any case, both approaches are valid and complementary.

I fear that frontier AI is going to be nationalised for military purposes, not just in the US but across the globe.

At the same time, I really don’t know what Anthropic were expecting when they described their technology as potentially more dangerous than an atom bomb while agreeing to integrate purpose-built models with Palantir to be deployed in high-security networks for classified military tasks.