A brain cut from its body and frozen its a dead brain.

It is like a crystal that shows beautiful colours when you shine a light through it. You can play with different kinds of lights and patterns, or you can put it in a drawer and forget about it: the crystal doesn’t care anyway.

Bundle of tokens comes in, bundle of tokens comes out. If there is any trace of consciousness or subjectivity in there, it exists only while matrices are being multiplied.

However, that number would typically be very very very very small, making it hard to argue that the whole model is a derivative of that one individual document.

Nevertheless, a similar approach might work if you took a FOSS project as a whole, e.g. "the model knows a lot about the Linux kernel because it has been trained on its source code".

However, it is still not clear that this would be necessarily unlawful or make the LLM output a derivative work in all cases.

It seems to me that LLMs are trained on large FOSS projects as a way to teach them generalisable development skills, with the side effect of learning a lot about those particular projects.

So if I used a LLM to contribute to the kernel, clearly it would be drawing on information acquired during its training on the kernel's code source. Perhaps it could be argued that the output in that case would be a derivative?

But if I used a LLM to write a completely unrelated piece of software, the kernel training set would be contributing a lot less to the output.

> “Good at explaining requirements, needs handholding to understand complex algorithms, picky with the wording of comments, slightly higher than average number of tokens per feature.”

I’m not saying this would be good at all, but the data (/insights) and the opportunity are clearly there.

If the agent comes back in a few minutes with a tiny fix, it is probably a small task.

If the agent produces a large, convoluted solution that would need careful review, it is at least a medium task.

And if the agent gets stuck, runs into architectural constraints, etc. then it is definitely a hard task.

In practice, arriving at this ideal scenario can be very challenging. Actually feasible experiments will be necessarily narrow, with the expectation that their results can be (roughly) extrapolated outside of their specific experimental setup.

Another valid approach would be to carry out qualitative research, for example a case study. This typically requires the study of one (or a few) developers and their specific contexts in great detail. The idea is that a deep understanding of how one person navigates their work and their tools would provide us with insights that might be related to our specific situation.

Personally, in this particular area, I tend to prefer detailed qualitative accounts of how other developers are working on similar projects and with similar tools as me.

But in any case, both approaches are valid and complementary.

I fear that frontier AI is going to be nationalised for military purposes, not just in the US but across the globe.

At the same time, I really don’t know what Anthropic were expecting when they described their technology as potentially more dangerous than an atom bomb while agreeing to integrate purpose-built models with Palantir to be deployed in high-security networks for classified military tasks.

My understanding is that Anthropic requested visibility and a say into how their models were being used for classified tasks, while the DoD wanted to expand the scope of those tasks into areas that Anthropic found objectionable. Both of those proposals were unacceptable for the other side.

(It’s a big if! I’m not convinced about that myself, but it’s worth considering that possibility.)

The Mayan and the Aztecs were roughly at a similar level of development as ancient Sumer or Babylon: good agricultural practices, irrigation, astronomy, elaborated culture, rich mythologies, very basic metallurgy, early state structures, etc.

Sumer and Babylon were great civilisations whose legacy can still be traced today. The same is true for the Maya and the Aztec. Had you visited any of them in their prime, you would have been awed by their skill and sophistication.

And yet, think of everything that happened in Eurasia between Hammurabi and Columbus, and you will get a sense of how wide the gap was when the two worlds met.

China is currently providing weapons for Russia to wage large-scale war in Europe, while supporting a dictatorship in North Korea that regularly launches nuclear-capable misiles in my general direction.

Historically, it would seem that often lowering the amount of people needed to produce a good is precisely what makes it cheaper.

So it’s not hard to imagine a world where AI tools make expert software developers significantly more productive while enabling other workers to use their own little programs and automations on their own jobs.

In such a world, the number of “lines of code” being used would be much greater that today.

But it is not clear to me that the amount of people working full time as “software developers“ would be larger as well.