Hacker News: gertlabs

New comment by gertlabs in "GLM-5.2 is the new leading open weights model on Artificial Analysis"

gertlabs — Thu, 18 Jun 2026 04:43:31 +0000

They're within confidence intervals of each other, but remember how much discussion there was that Opus 4.6 had been nerfed in March. We averaged samples over the entire lifetime of Opus 4.6, which likely served many different underlying checkpoints. Even the best version of Opus 4.6 was hardly an upgrade.

We find a lot of interesting anomalies with our benchmark that hold up under large sample sizes.

New comment by gertlabs in "A robot is sprinting towards you. Do you want it running on Claude or Grok?"

gertlabs — Wed, 17 Jun 2026 22:41:46 +0000

All of our posts have been well received by an insanely high percentage of people who have interacted on here -- most people clearly find what we're doing interesting and relevant to the HN community (AI evaluations). A flag seems pretty aggressive! Especially when the top comment on the article (after our above comment got flagged) is about tacos.

I'm a person running the account, and I only post where I think we have a relevant contribution.

New comment by gertlabs in "GLM 5.2 Performance Benchmarks"

gertlabs — Wed, 17 Jun 2026 16:52:10 +0000

On our multi-agent coding and reasoning evaluations, GLM 5.2 is the first model we've tested that crossed the threshold of being on par with or better than Opus 4.6 (although as usual, we have GLM 5.2 and most other Chinese models a bit below most other benchmarks with test methodologies that are more vulnerable to benchmaxxing).

Data at https://gertlabs.com/rankings

New comment by gertlabs in "GLM-5.2 is the new leading open weights model on Artificial Analysis"

gertlabs — Wed, 17 Jun 2026 16:43:44 +0000

GLM 5.2 is the first model we've tested that is unambiguously on par with, or better than Opus 4.6 (although as usual, we have GLM 5.2 and most other Chinese models a bit below most other benchmarks with more vulnerable test methodologies).

Data at https://gertlabs.com/rankings

New comment by gertlabs in "MiMo-v2.5-Pro-UltraSpeed: 1T model with 1000 tokens per second"

gertlabs — Tue, 09 Jun 2026 01:51:15 +0000

It's likely overfit to common harnesses and iteration patterns, so it struggles with formatting tool calls and json in our testing which use our own harnesses (although there is a lot of overlap with tools that would be found in any coding harness like bash, apply_patch, etc.)

We didn't love the results because it draws negative scrutiny to our benchmark, but the results are real and done at scale and I think DeepSeek V4 Pro's inability to do agentic work outside of environments it was trained on is an important thing to measure, especially when so many other models can generalize to new environments just fine.

Google models also struggle with tools, but they have very strong initial answers, so there is more potential for them to bridge the gap with some better post-training.

New comment by gertlabs in "MiMo-v2.5-Pro-UltraSpeed: 1T model with 1000 tokens per second"

gertlabs — Mon, 08 Jun 2026 19:59:39 +0000

DeepSeek v4 Pro struggles with a custom harness, and all the models ranked above it don't, so it gets downweighted in the agentic coding benchmarks (although it ranks better than Flash in one-shot problem solving: https://gertlabs.com/rankings?ow=1&mode=oneshot_coding). We ran plenty of samples.

MiMo v2.5 is on there, as well as the pro version.

We found a few anomalies in our evaluations, which makes sense -- if every new sub-release is better across the board in every area of the model card, that should raise alarms about benchmaxxing. But the main thing we found is that hype != performance, and I trust our benchmark methodology significantly more than the model cards the labs add to their press releases.

New comment by gertlabs in "MiMo-v2.5-Pro-UltraSpeed: 1T model with 1000 tokens per second"

gertlabs — Mon, 08 Jun 2026 16:44:13 +0000

MiMo V2.5 Pro (regular speed) remains the strongest open weights agentic coding model we've tested -- it's been interesting to see how little attention it has received relative to some lower performing releases. And the "fast mode" pricing is very competitive here.

Data at https://gertlabs.com/rankings

New comment by gertlabs in "My thoughts after using Clojure for about a month"

gertlabs — Tue, 02 Jun 2026 22:38:37 +0000

GPT 5.4+ models are extremely good at writing Clojure, agreed. In the agentic coding part of our benchmark, they do have access to the REPL via bash if they choose to use it. Filtered here: https://gertlabs.com/rankings?mode=agentic_coding

New comment by gertlabs in "My thoughts after using Clojure for about a month"

gertlabs — Tue, 02 Jun 2026 22:34:58 +0000

I might just be a simpleton -- I never had the resolve to try an ambitious project in Clojure. I was not aware that you could get full OOP though, what you are describing feels like yes technically possible but kind of a hack to get inheritance / no type hierarchy enforcement. I'm no expert on the language though

New comment by gertlabs in "My thoughts after using Clojure for about a month"

gertlabs — Tue, 02 Jun 2026 22:28:35 +0000

Success rate includes syntax/compilation failures as well as environment rule violations, and is almost entirely from one-shot code generations. Percentile shows how well the working submissions perform.

In long horizon agentic coding evaluations, strong models fix the syntax and percentile and it becomes a direct comparison of which submissions per language performed the best on average. You can filter for that here: https://gertlabs.com/rankings?provider=openai&mode=agentic_c...

New comment by gertlabs in "My thoughts after using Clojure for about a month"

gertlabs — Tue, 02 Jun 2026 22:02:38 +0000

The functional paradigm is a bit uncomfortable at first, but it does make problem solving feel... different. I personally find OOP to be the most intuitive for large scale systems design, but that's just me.

Most models do not perform particularly well in Clojure, but OpenAI models fully utilize the power of the language. Subjectively, it kind of seems to match the personality. Data at https://gertlabs.com/rankings?provider=openai

New comment by gertlabs in "Social Intelligence Benchmark"

gertlabs — Tue, 02 Jun 2026 07:52:09 +0000

Nice, that's a good one -- interesting dynamics can come out of deceptively simple social games.

Social Intelligence Benchmark

gertlabs — Tue, 02 Jun 2026 06:52:37 +0000

Article URL: https://gertlabs.com/blog/social-intelligence-benchmark

Comments URL: https://news.ycombinator.com/item?id=48366885

Points: 5

# Comments: 2

New comment by gertlabs in "OpenRouter raises $113M Series B"

gertlabs — Sat, 30 May 2026 18:42:26 +0000

OpenRouter is our primary provider for evaluation data, and we've been really happy with them!

I'm sure they're experiencing growing pains, but a larger model selection (and faster releases for open weights models), would keep us from using other providers. For example, it took much longer than it should have to get Qwen 3.6 ~30B class models released (almost 2 weeks if I recall)

New comment by gertlabs in "LLM Paper Trading"

gertlabs — Sat, 30 May 2026 07:35:01 +0000

That's an interesting point -- they are told that they are paper trading. Maybe we should run another session that A/B tests this.

New comment by gertlabs in "LLM Paper Trading"

gertlabs — Sat, 30 May 2026 07:18:14 +0000

We're a month into a long running experiment to see how recent models perform in day trading, where they have a constant harness giving them the ability to write code, access the web, take notes, and install handlers to trade for them. Realistic slippage and fees and margin requirements are built into the simulation.

The harness and the capabilities built into the environment gives the models all the resources they would realistically ever have access to in order to help them succeed (or fail).

We just tell the models to maximize portfolio balance in the long term. The models are responsible for any good (or bad) ideas.

LLM Paper Trading

gertlabs — Sat, 30 May 2026 07:18:14 +0000

Article URL: https://gertlabs.com/spectate?game=trading

Comments URL: https://news.ycombinator.com/item?id=48333564

Points: 6

# Comments: 5

New comment by gertlabs in "Notes from the Mistral AI Now Summit in Paris"

gertlabs — Fri, 29 May 2026 19:01:43 +0000

There is likely a theoretical limit to how much intelligence you can pack into a model of a given size (especially when stretching that over a large input context size).

Our evals are pretty complex so we only recently started testing ~30B class models, which are now becoming quite smart (on par with the frontier from 1 year ago). Mistral is far behind, but I'm rooting for them.

Data at https://gertlabs.com/rankings

New comment by gertlabs in "Claude Opus 4.8"

gertlabs — Fri, 29 May 2026 14:28:37 +0000

That'll populate over the next couple weeks -- those are the live games on the spectate tab which take a while to generate statistically worthwhile data. I'm curious how it does. From using it all day, I can say Opus 4.8 is my new favorite model, hands down.

New comment by gertlabs in "Claude Opus 4.8"

gertlabs — Fri, 29 May 2026 03:58:50 +0000

Appreciate that! Results are live: https://gertlabs.com/rankings

Opus 4.8 is the first tangible improvement since Opus 4.5. And it doesn't seem to have the personality problems of the last release -- I've been enjoying using it.