I've spent probably over100 hours working on this benchmarking/site platform, and all tests are manually written. For me (and many others that reached out to me) are not useless either. I use this myself regularly when choosing and comparing new models. I honestly beleive it is providing value to the conversation.

Let me know if you know of a better platform you can use to compare models, I built this one because I didn't find any with good enough UX.

I think the model is now tuned more towards agentic use/coding than general intelligence.

[0]: https://aibenchy.com/compare/z-ai-glm-5-medium/z-ai-glm-5-1-...

I assume it is the 26B A4B one, if it runs locally?

I ended up switching to plain PHP, and it worked great. It is still mostly "static", but I can dynamically include the same content on multiple pages without having to duplicate/build it every time.

1. Most variants on HIGH/XHIGH provide only marginal improvements in accuracy, but at drastically increased latency and cost. One special example is Gemini 3.1 Flash Lite, which on High used 1.5M reasoning tokens, and it's cost was 5x the one of running 5.3-Codex: https://aibenchy.com/compare/google-gemini-3-1-flash-lite-pr...

2. On medium it seems like most models use a similar amount of reasoning tokens, this should be a more fair comparison.

3. Most models in the wild are used on medium (chat apps, default coding apps, tools, etc.).

4. Running on models on HIGH/XHIGH can lead to huge costs for me maintaining the test suite. I might add more models on high, if I can do it in a sustainable way.

5. Running models on HIGH would make running tests suites take much longer, so the results won't be published as fast.

6. Some models even show degradation when used on HIGH, as they tend to overthink/doubt themselves more. This seems to be a trend especially for new models, which wore trained to actually say "wait, but" quite a lot...

Overall, I am happy with how the current leaderboard/comparisons work. I might test some models on high, but for me, a better indication of true intelligence of a model/AGI is how well it does with "none"/no reasoning, than how well it does with high.

https://aibenchy.com/compare/google-gemma-4-31b-it-medium/go...

I think GLM got a bit in front, because on some tests that both got wrong, GLM did sometimes (inconsistently) respond with the correct answer.

That being said, yes, in this case probably with more and more tests added, gpt-5.4 would edge in front, especially if a coding would be added (there are no coding tests yet).

Note that all reasoning models are tested with "medium" reasoning.

The benchmarks are questions/data processing tasks that an average user will likely ask, not coding questions (I didn't add any coding tests yet).

Gemini models also tend to be very consistent. Asking the same question will likely give the same result.

The two models you mention scored the same, the only difference is that Gemini was better at domain-specific questions (i.e. you ask something quite technical/niche).

Even when using structured output, sometimes you want to define how the data should be displayed or formatted, especially for cases like chat bots, article writing, tool usage, calling external api's, parsing documents, etc.

Most models get this right. Also, this is just one failure mode of Claude.

The problem with doing this is cost. Constsntly testing a lot of models on a large dataset can get really costly.

https://aibenchy.com/compare/minimax-minimax-m2-7-medium/moo...

A lot of alerts, reporting, quotes, code snippets, short documentation or step by step instructions, etc.

I don't just send emails to say "Hey, let's meet at 5". You know the memes with "this could have been an email", it usually is this case.

Just to be clear, most of those rich emails are the automatic/transactional emails.