Flexibility and abstraction come at a high cost. It doesn't really matter though, world domination at all costs is the name of the game.

That said, whether the game itself will be good is still TBD.

Bolting on new data to an existing model (fine-tuning) is precisely what gets you catastrophic forgetting.

Sure an LLM can update every table in your project every time you make a change, just as a human could. It also can make mistakes and burn significantly more tokens doing so. Or, it could update one component with a one line change and be done. Easy to verify, less change for mistakes.

Engineering is a discipline and does not change dramatically with the advent of outsourcing it to agents.

Because backprop fundamentally wants the entire data set in every pass. It doesn't behave well and is destructive when you update after the pre-training phase. RLHF/LORA are attempts to work around that and effective at what they do, but it is not learning in the sense you are talking about and also do not fully address the catastrophic forgetting problem. This architecture as is - is not compatible with continual learning.

What prevents continuous fine-tuning from what I understand is catastrophic forgetting. You can do things like RLHF which are built to minimize the damage but that is more about bringing out capabilities of the base model than incorporating new knowledge (at least from my understanding, I am obviously not a researcher at a lab).

To me the heart of the "next token predictor" is that the distributions are static. You can manipulate what you feed into it through context (and a lot of interesting engineering has been applied there through CoT and other techniques to manipulate the prompt). But these models as implemented will never be able try things and learn from mistakes or adapt. They are a set of weights frozen in time. A set of distributions derived from the original data that created them.