There would be so many examples of coding projects that these models began or attempted to work in, that were abandoned because the models were floundering.

I would imagine the labs have some decent ways to produce novel requirements and then actually validate they are met, without the noisiness of implicit human feedback.

That said, the more I think about it, you are right, there's probably also very good ways to extract signal for all these sessions.

This kind of raises another question to me regarding the coding benchmarks, how much of it is model versus harness?

Wouldn't surprise me if in the next few generations we start seeing more LLM generated languages that LLM's prefer to use for expressability, conciseness, etc.