Through training on human text, we are building implicitly in the weights a statistical model of what humans might write in response when presented with arbitrary pieces of text. It turns out that we can make these incredibly accurate.

If building an accurate internal model of something then using it to predict that thing’s behaviour is different to gaining understanding of that thing, we will need to pin down exactly what “understanding” means, or we are forever doomed to talk at cross purposes.

Turing complete systems can be built out of matrix multiplications, out of attention, out of key/value lookups. The Chinese room is Turing complete. By claiming it cannot understand things because it is built out of components computing devices can be built out of, we are claiming no computer can because no computer can. This is a very bold claim indeed, and also we’re assuming the conclusion! The claim is no more convincing than “brains cannot understand things because they are made out of neurons”. The system may or may not have some particular properties, but we have to do more work than just gesturing at the components the system is made of when making claims about it; the alternative is, at best, a world where we prove too much and conclude that humans, too, are not conscious.

For starters, we need to pin down the terms under discussion enough that they don’t just mean whatever we need them to in the moment.

...so, a software license.

This isn’t usually a required engineering skill. I’m guessing the interview was designed for salespeople and/or middle management.

Some would prefer less excitement than this.

C code should be more predictable and easier to reason about than using a macro assembler. To the extent it is not, the language has failed.

A lot of UB should be implementation defined behaviour instead; this would much better match programmers’ intuitions as they reason about their code - you can even see it in the comments of this post: it’s always things like “this hardware supports / doesn’t support unaligned accesses”, it’s never nasal demons.

Guess at the end of the day, no-one ever got fired for building ARM.

For me, the line is precisely at the point where a human has something they want to say. IMO - use the tools you need to say the thing you want to say; it's fine. The thing I, and many others here, object to is being asked to read reams of text that no-one could be bothered to write.

Any scheme that doesn’t require this won’t get pushback from me.

As an alternative: I already have government-issued ID and that branch of government already has my private info; have it give me a cryptographic token I can use to prove my age bracket to the root of trust module in my computer; then allow the OS to state my age to third parties when it needs to with a protocol that proves it has seen the appropriate government token but reveals nothing else about my identity.

Other alternatives are possible.