When a task is initiated, it starts from a need, from a specific context. To work it out the AI needs to continuously interact with the context, and get feedback from it. At the end gains, losses, risks and costs sink back in the context.

The context is you, the person who prompts, your team or company. It is indexical and relational. It is maximally distributed. It cannot be hoarded. You can't eat so that I feel satiated. AI is called to do the work, but it can't handle 3 things - start, middle and end of a task.

Qualia represents the compressed past experiences acting as a screen on which we represent new experiences, language is compressed past experiences from others and from past generations. Both work to reduce costs of cognition and action. (imho)

We also know brains are locked inside a bone box only connected to the outside world by a bundle of unlabeled nerves, there is no direct access. So the brain can only compare patterns of signals it receives from outside. But since this representation-action-learning loop is recursive it cannot be inhabited or known from outside, 3p needs to pay the price of recursion to execute in order to get to 1p.

The gap is that between description and execution, which cannot be crossed for free with cheap description. Execution costs, and that cost is part of what is like being a bat. We can't inhabit their cost pressures since we don't have their context and body. You can't remove the costs of being a bat from "what it is like being a bat" and still get your answer from the comfort of the philosophical armchair.

"Evaluation" means environments or datasets, the model is supposed to discover its representations from scaled up experience. That was the bitter lesson - more data and compute beat heuristics.

There is research in open-ended learning, see "Why Greatness Cannot Be Planned" by Kenneth O. Stanley. The core idea is that in open-ended scenarios you don't know what action was good except in hindsight because your path is deceptive. So the idea is to replace fitness with novelty search which provides more stepping stones towards the goal.

That is a problem in RL, so we usually do supervised training first, teach it to imitate some trajectories, then do RL to refine the model. RL alone has a huge problem because it might be hard to reach a reward, hence hard to learn the task by pure reinforcement. Humans also combine supervision (learn from books) with search (solving problems) to break the discovery problem. For example, a human with no initial instruction in math would not produce great results no matter how smart they are. The bootstrap was exploration paid for in the past.

If you extract the spec from first implementation and reimplement from scratch you get a free testing oracle. Where they diverge you send the agent to decide which one had a bug.

In ML it was even worse, we had to throw away a decade of experience, made irrelevant by the new approaches. Even the most revered activity - designing new architectures - became too expensive to do in real life. Fine-tuning models is what we do now, prompting and evals. Like 90% of what we used to learn is no longer needed. And yes, LLMs can do most new ML activities too, they just need light supervision. I am sometimes ashamed to admit I have stopped coding 12 months ago and never wrote one more line, that after 35 years of coding manually. But I also think we will never be without LLMs again, so no point in preparing for 2016 in 2026

I think on the contrary, LLM providers accumulate huge logs of interaction with their users, which elicit that tacit knowledge and mine it and humans cooperate willingly in order to solve their tasks. Just imagine the corpus of sessions for scientific research, education or software development, it is probably the largest such collection ever to exist. Trillions of HITL tokens per day flow into those logs, carrying our perspectives, choices, original ideas and tacit knowledge. I call this the "human-AI experience flywheel". It's the new stackoverflow, next model generation is based on interaction data from previous one.

Imagine you have a 4B model and keep an equal size corpus of search stubs, small MD documents linking to hubs, feeds and search engines for millions of topics. You can use the LLM to read the stub and perform the search for you, all orchestrated locally, with greater privacy and independence. You can dis-intermediate the search chokepoint now. You can set the criteria for what to include, exclude, how to rank and present the results.

This works because good entry points for any topic change slowly over time. The construction of search stubs is trivial with existing AI agents, and can be shared as open source. A few GB for the model, a few for the search routing layer, and you got a sovereign local agent.

If this holds, access control shifts from whatever Google thinks maximizes profit to whatever the community thinks has value.

It does work, it is called RLVR, reinforcement learning from verified rewards, is is based on testing code by execution. It's become a major area of improvement in the last year. But you are also forgetting the amount of steering and problem solving going into coding agents today, and the huge logs they create which can feedback into training. We automated stackoverflow, LLM learns from usage and self play.