tldr 2% average point lost on Rust compared to python, gap vary by model, go has a better upper bound but opus had it 3% below python.

benchmark is a bit old but research on why is there, article is just vibes

the AGENTS.md pieces that pin specific tool-call shapes or force chain-of-thought before action are coping that ages out, same lifecycle as the retry-with-different-prompt loops or chains of thought prompt most stacks shipped in 2024 to compensate for brittle instruction-following.

not quite there yet, but it's nice to see them being shorter and shorter as model release until all the basic are peeled out by the march of progress and one day only the invariants will be left there

that said context management seem to be solving today model problems, more than being an universal property, and will probably be obsoleted a few model generations down the road, as tool obsoleted RAG context injection from question embeddings.

Most agent stacks at AI startups have that layer as llm driven glue rather than an owned surface, and it shows up as a re-architecture cost on every model release. model should be replaceable, the integrations and guardrails specific to the customer's environment should not.

but yeah you're correct anyone optimizing for public-bench rank instead of their own task-distribution eval has been pointing at the wrong thing for a while

still I guess useful signal to know which one model to consider, negative signal is still signal, assuming everyone is gaming benchmark in certain ways, lack of performance do result in a real workload effect

I mean beyond the obvious hacker news bias.

If you like it nobody will remove it to you as a hobby. But the artisanal aspect of coding as a production mechansim is dying, and it was about time.

Not saying it is better or worse, but the way I perpersonally prefer is to design in chat, to make sure all unknown unknown are addressed

That why vc look favorably to startup which go trough the motion of setting up partner led sales channel. an established partner taking maintenance contracts bridge the disconnect in the lifecycle gap between the two realities.

But no, corporate is bad, I guess.

So while I agree mixed model is the way to go, opus is still my workhorse.