This is more of a faster paced/short lived game so we can collect larger samples of data on larger groups to get significant results in model behaviors of collaboration, truth telling, and ability to lie effectively.

In the leaderboards part of the page I'll be autopopulating the token cost of the model as a metric to evaluate on

I built ClashAI to be an open agent scoreboard where frontier models play against each other in environments like Civilization and other strategy games. Every match is streamed live with the AI thinking fully observable.

The agent rankings will be continually updated and reflected as we add environments.

Brief notes on CivBench Season #001: - 200 turn limit

- Starting with 8 of the top 42 agents we’ve tested in a standardized harness

- 90s reasoning timeout (timed with thinking config per model card)

- live benchmark, still growing sample size

What’s been interesting so far:

Models that look similar on static benchmarks can diverge meaningfully in long-horizon matches. In early CivBench runs, we see distinct strategy tendencies (e.g., military-forward vs economy/tech-first openings), plus clear differences in execution profile (latency, token cost, actions per turn). In some matchups, lower-cost models move through turns faster while remaining competitive on outcome metrics.

Some measuring notes: - test runs are expensive for max configurations, running Claude Opus 4.6 cost us $1200 one match. We tuned accordingly - sometimes LLM providers are flaky/slow even though their models are fast.

If you’re looking to access the data as a research team or interested in hosting an environment please get in touch!

Thanks to the OG freeciv community

LINKS:

freeciv-llm: https://github.com/taso-ventures/freeciv-llm

Initial learnings: https://www.clashai.live/blog/ai/introducing-civbench-season...

Comments URL: https://news.ycombinator.com/item?id=47152571

Points: 12

# Comments: 24

The commenter who says "add obfuscation and success drops to zero" is right but that's also the wrong approach imo. The experiment isn't claiming AI can defeat a competent attacker. It's asking whether AI agents can replicate what a skilled (RE) specialist does on an unobfuscated binary. That's a legitimate, deployable use case (internal audit, code review, legacy binary analysis) even if it doesn't cover adversarial-grade malware.

The more useful framing: what's the right threat model? If you're defending against script kiddies and automated tooling, AI-assisted RE might already be good enough. If you're defending against targeted attacks by people who know you're using AI detection, the bar is much higher and this test doesn't speak to it.

What would actually settle the "ready for production" question: run the same test with the weakest obfuscation that matters in real deployments (import hiding, string encoding), not adversarial-grade obfuscation. That's the boundary condition.

For example, I'm sure an RL bot will be able to figure out an optimal strategy over millions of simulations that defeats current LLMs with context, however this may not always hold true