This trend has been there since we started evaluating models using different languages in February 2026 and if anything, the disparity has grown in frontier models. Even Google models prefer Kotlin/C#/Rust for coming up with creative ideas (compilation success is a different story). Data at https://gertlabs.com/rankings

That being said, models love to recommend Go, and Go does have a lot going for it, especially if you are serving a public-facing website. So most of our public facing API handlers are written in Go, and we offload some of our most important binaries to Rust. There are just too many reasons not to use the languages that models think a little more effectively in.

So just adding a language or tag filter can result in some pretty small sample sizes. You can see how many samples survived in the box plot, but that's probably bad UX that most people never see. There's a reason no other benchmark provides this type of data (even for our sample sizes it runs almost 10K USD/month to keep up to date with new releases).

Might be a good idea to reduce the ability to apply filters into a cohort with less than ~20 samples -- not the first time we've gotten that feedback. Seems like adding too many options to see individual sample variation is just misdirecting. I'm a nerd who loves data so I hate removing access, especially since the aggregate performance is very interesting (averaged across all languages, we see consistent and interesting performance data across models, like models outperforming with strongly typed languages). But tbh I think you're right and we'll try limiting filters to where we actually have statistically significant data.

I agree that Opus 5 is not a great model, despite being clearly intelligent. It seems like a personality problem in user-driven agentic coding workflows, not a real capability issue. Not incorporating unspoken user intent, going off topic, incorporating some of the pedantry you find in GPT 5.x models, etc.

That's also likely why Opus 5 ranks low on our "Social Intelligence" benchmark (https://gertlabs.com/rankings?mode=decision), although sample sizes on this one are still low.

Data at https://gertlabs.com/rankings

All models have probably memorized significant swaths of solution sets for popular benchmarks at this point, either accidentally or intentionally, so it's all relative at this point. However, in our experience, Chinese models do benchmax harder. This is also consistent with interacting with Chinese labs soliciting data/environments, who literally asked us for datasets and tasks modeled around and formatted like popular benchmarks.

Opus 5 will be uploaded tomorrow, but we already have the tests locally and it is truly as capable as Fable, but at 81% of the real cost. (And from subjective usage, it has a very different personality)

Data at https://gertlabs.com/rankings

We run an evaluation that only compares models in open-ended multi-agent environments where agents affect each other, primarily testing writing code. It's designed to be less vulnerable because there's no solution set, and it's been pretty effective and tends to rank Chinese models lower than their advertised model cards (relative to US models). Kimi K3 is a bit of an exception there -- it truly is a near-frontier model. But it's so slow. Ironically, Muse Spark 1.1 is one of the strongest models we've tested after Fable and Sol while also leading the cost efficiency curve. Big turnaround from Llama 4.

Data at https://gertlabs.com/rankings

Comments URL: https://news.ycombinator.com/item?id=48975007

Points: 2

# Comments: 0

Agentic coding is what's most relevant to software engineers, but so is speed, which is a real usability issue right now. Third-party inference providers like Fireworks have bridged the gap for some previous releases.

It's exciting to see the open weights frontier becoming the norm. Competing on having the frontier model is going to be an increasingly difficult business.

Data at https://gertlabs.com/rankings?mode=agentic_coding

I have not broken down the comment/code ratio, but that's actually a really interesting idea for a metric.

I would also like to test Cursor, but our policy is to only test models available on public routers for now.

Agentic coding data: https://gertlabs.com/rankings?mode=agentic_coding

However, the fact that they finally have a strong post-training and RL setup bodes well for future releases. They certainly are not compute-constrained anymore.

Data at https://gertlabs.com/rankings?mode=oneshot_coding

Fable's main advantage is that its average solution size is smaller. However, GPT 5.6 Sol is a substantial improvement from GPT 5.4/5.5 which would write verbose, defensive code. 31KB for GPT 5.4/5.5 down to 26KB for GPT 5.6 Sol, with better performance for Sol.

Fable scores slightly lower, but with an average solution size of 12.2 KB.

Data at https://gertlabs.com/rankings?mode=oneshot_coding

Sonnet 5's performance is comparable to GLM 5.2 in both one-shot coding and agentic ability. However, it's about ~20% less verbose than GLM 5.2 in average code submission sizes, and uses fewer reasoning tokens, which reduces the cost gap and suggests it writes cleaner code. In practice, Sonnet 5 ends up being 40% more expensive and ~2x faster than GLM 5.2 in our evaluations (not 300% more expensive as the per-token pricing would suggest). Granted, GLM 5.2 is an extremely reasoning heavy model.

Overall, it's a solid release that gives Anthropic some standing in the price-conscious inference market.

Data at https://gertlabs.com/rankings

It doesn't have a higher capability score than Fable, though. We break our coding evaluations into 2 parts, and "one-shot coding" makes up part of the index, where Fable significantly outperforms every other model, which is why it's ranked at the top despite Sonnet 4.6 having a slightly higher median (and lower average) in long-horizon agentic workloads. One-shot coding tends to be the most correlated with other companies' model cards, whereas agentic coding is partly about how well a model can adapt to a custom harness. Fable also refused some tasks.

Data at https://gertlabs.com/rankings?ow=1&mode=oneshot_coding

We now use Sonnet 4.6 for a number of internal use cases we wouldn't have considered otherwise.