I have shared examples that the tested AI models have written. It turns out that the AI models write code that does not resemble any of the bots in the training data.

https://github.com/llmskirmish/starskirmish-hillclimb

Vals IOI, olympiad programming problems solved in C++, and WeirdML v3, unusual machine learning coding tasks also mirror StarSkirmish Bench.

I did a StarCraft: BW hillclimb experiment with GPT-6 Astra and Claude 5.5 Opus. Peter Gostev did a Chess hillclimb saw similar behavior as I did with GPT-6 Astra.

I am considering an experiment where the task is to try to win by adversarially targeting Pluto (a high profile self play based RL model)

Protoss vs Protoss only also helps downscope the problem given the time constraints

There's also a pretty high profile StarCraft: BW bot right now called Pluto trained exclusively by self play RL (AlphaStar style).

I did a 48 hour test with GPT-6 Astra and Claude Opus 5.5 and we saw some interesting uses of Psionic Storm.

I've open sourced some of GPT-Astra's work (below). It actually did an interesting amount of changes in strategy over time as it's opponents became more difficult.

https://github.com/llmskirmish/starskirmish-hillclimb

They drop to single digits against the A tier and S tier bots because those bots are just generally harder opponents to beat.

Although I should caveat the format advantaged the LLM bots compared to traditional tournament rules.

Comments URL: https://news.ycombinator.com/item?id=50118688

Points: 249

# Comments: 111

Comments URL: https://news.ycombinator.com/item?id=49943964

Points: 6

# Comments: 0

Each LLM writes a Protoss bot in C++ against BWAPI, played on OpenBW. They must climb five tiers of Protoss opponents, from scripted demo bots up to top rated human written bots like Stardust and PurpleWave.

The StarSkirmish bench restricted LLMs to 1 hour of wall clock time for reasoning. With Hillclimb there is no fixed time limit.

GPT-6 Astra and Claude 5.5 Opus both benefit from substantially longer reasoning periods than 1 hour and so this format is a more realistic depiction of their abilities

Comments URL: https://news.ycombinator.com/item?id=49935608

Points: 2

# Comments: 0

It draws on my previous work, LLM Skirmish, where LLMs were asked to write JavaScript strategies to compete in a Screeps like RTS.

In StarSkirmish, LLMs get 1 hour of wall clock time to write the strongest StarCraft BW bots they can in C++. It is then evaluated against other LLMs' bots and established human written bots.

All bot are BWAPI 4.4.0, played via OpenBW. All scored matches are Protoss vs Protoss only right now.

GPT-6 Astra and Claude Opus 5.5 are functionally tied as the top 2 scoring StarSkirmish Bench LLMs.

Most interestingly to me is StarSkirmish suggests a much more precipitous drop off in capability's once you go outside the OpenAI and Anthropic frontier, then many other public benchmarks would suggest.

Comments URL: https://news.ycombinator.com/item?id=49858284

Points: 4

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=49849117

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=47786753

Points: 3

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=47605879

Points: 2

# Comments: 0

In the benchmark, in each round every LLM plays every opponent, and then we do that multiple times (an "epoch").

In the community ladder, when a player submits a strategy it plays a match against the latest strategy submitted by every player.