--- title: Show HN: Gemini 4 Argon plays StarCraft | daily.dev description: StarSkirmish Bench v0.2 measures how well flagship LLMs (GPT-6 Astra, GPT-6.1 Sol, Gemini 4 Argon, Claude Opus 5.5, Claude Fable 5.1, Muse Wonder Full, and... canonical: https://daily.dev/posts/show-hn-gemini-4-argon-plays-starcraft-1dpc339s0 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Show HN: Gemini 4 Argon plays StarCraft | daily.dev og:description: StarSkirmish Bench v0.2 measures how well flagship LLMs (GPT-6 Astra, GPT-6.1 Sol, Gemini 4 Argon, Claude Opus 5.5, Claude Fable 5.1, Muse Wonder Full, and... og:url: https://daily.dev/posts/show-hn-gemini-4-argon-plays-starcraft-1dpc339s0 og:image: https://api.daily.dev/og/posts/1dPC339S0.png og:image:alt: Show HN: Gemini 4 Argon plays StarCraft og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Show HN: Gemini 4 Argon plays StarCraft **[Hacker News](https://daily.dev/sources/hn)** · 8 min read · 0 upvotes · 0 comments ## Summary StarSkirmish Bench v0.2 measures how well flagship LLMs (GPT-6 Astra, GPT-6.1 Sol, Gemini 4 Argon, Claude Opus 5.5, Claude Fable 5.1, Muse Wonder Full, and others) can write a Protoss StarCraft: Brood War bot in C++ within one hour, each running in its own lab's first-party CLI harness at maximum reasoning effort. Bots are scored by win rate against a fixed roster of competitive human-written bots and demo bots rather than Elo, since v0.1. GPT-6 Astra tops the field, with GPT-6.1 Sol a clear second and offering far better cost efficiency; Gemini 4 Argon, Claude Opus 5.5, and Claude Fable 5.1 cluster behind. Even the best model wins only 7% of games against the strongest human bots (Stardust, PurpleWave, BananaBrain), showing the benchmark is far from saturated. The strongest predictor of score is how much a model practices against strong bots during its hour, and results correlate weakly with popular benchmarks like Artificial Analysis Intelligence Index but more closely with Vals IOI and WeirdML v3, suggesting heavily targeted benchmarks may be overfit. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Which LLM performs best at writing a StarCraft bot in one hour on StarSkirmish Bench v0.2 GPT-6 Astra scores highest on StarSkirmish Bench v0.2, a clear step ahead of GPT-6.1 Sol in second place. Gemini 4 Argon, Claude Opus 5.5, and Claude Fable 5.1 are functionally tied behind them, scoring between 37 and 39. GPT-6.1 Sol offers the best value, scoring second at roughly a tenth of GPT-6 Astra's API cost per one-hour run. _Comparing flagship model performance and cost is easier when tracking benchmarks like this on daily.dev._ ### How well do the best LLM-written StarCraft bots perform against top human-written bots Even the best model, GPT-6 Astra, wins only 7% of its games against the strongest human-written bots Stardust, PurpleWave, and BananaBrain. This shows the benchmark is far from saturated despite rapid progress among models, with real tournament conditions against these bots being significantly harder than the three two-player maps used in testing. _Developers tracking how close AI coding agents get to expert-level output can follow benchmarks like this on daily.dev._ ### What is the strongest predictor of an LLM's score on StarSkirmish Bench The strongest predictor of a model's score is how much it practices against the seven strongest bots on the roster during its one-hour run. Top performers like GPT-6 Astra, GPT-6.1 Sol, and Gemini 4 Argon start playing these strong bots early even while losing, whereas weaker performers like Muse Wonder Full and Claude Haiku 5.5 rarely or never do. _Those evaluating agentic coding strategies can keep up with findings like this via daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 77 comments across hackernews (as of 2026-10-16). **TL;DR:** Reaction is mostly curious and analytical rather than celebratory: people like the end-to-end, time-boxed design but keep questioning what the benchmark actually measures and why top bots still lose badly to strong human-written opponents. **Sentiment:** 20% positive · 55% mixed · 25% skeptical **The case for** - The time-limited, win-rate-based format is seen as a refreshing contrast to open-ended or saturated benchmarks. - The site/presentation is praised as clean and free of typical 'AI slop' design. - Detailed play-by-play writeups (e.g. strategy shifts like proxy gates or Dark Templar openings) are considered genuinely interesting to dig into. - Correlation with other niche benchmarks (GoBench, Vals IOI, WeirdML) is noted as an intriguing, possibly meaningful signal. **The pushback** - Unclear what exactly is being measured since so many skills (coding speed, strategy, game understanding) are conflated into one score. - Skepticism that low win rates against human bots make sense if models can simply memorize/copy those same bots from training data. - Harness, tooling, and thinking-level differences across labs may be confounding the comparison more than raw model capability. - Doubts that the agent-written bots are doing novel reasoning rather than reproducing memorized open-source bot strategies. - Some question the choice of a 1-hour coding window and the Protoss-only scope as artificially narrowing what's being tested. **By community** - hackernews (mixed): Discussion is detailed and engaged, split between genuine interest in the benchmark's design/results and persistent doubts about what it actually measures and whether memorization explains the outcomes. **Hottest debate:** Whether the agent-built bots are demonstrating real strategic reasoning or just reproducing memorized human-written bot code, especially given their surprisingly low win rates against those same human bots. **Open questions** - Why do models with likely access to human bot code in training data still lose the vast majority of games against those bots? - How much of the performance difference is due to the harness/tooling/thinking-level setup versus the underlying model's capability? - Would results change if models could store/reuse data across games instead of starting fresh each time? - Is there meaningful insight to be gained from analyzing what agent-written bots actually do differently, beyond win/loss stats? **Highlights** > Adapted from what I wrote on this previously: https://x.com/dgant/status/2106880240907620731 The unsexy perspective on StarCraft is that it's a game about logistics. You need to keep making things, not get supply blocked, fit all your buildings, not get your units trapped, and then bring all your units to a fight at a good time and at the same time. Astra and Opus both cleared this bar pretty quickly, both settling on one-base Dragoon + Observer play, taking expansions pretty late, on 3 Gateways. This strategy is fine against everything and great against nothing, letting you win against anyone that you can out-execute. StarSkirmish gives agents access to the FastApproximation combat simulation library, which makes better combat predictions than the B-tier bots do, and let their bots win by trading armies better against them. But the A-tier bots (Locutus and BananaBrain) are already strong at execution, and Astra wasn't making progress against them. It switched to a proxy 2-gate Zealot rush. This worked well on a map pool with mostly 2-player maps (so you can place the Gateways very close to your opponent). But it's also an astute choice for a new bot because it doesn't require you to be good at anything in StarCraft that happens after the first four minutes! It's also smart as a variance-maximizing strategy. In many games, if your opponent is superior, you want the game to be decided by as few events as possible. Because with a bad coin, you're more likely to win 1 of 2 flips than 10 of 20. Proxy 2-gate shortens the game and the number of interactions required to win. It's also advantageous given the format, where you get unlimited tries to win a series against each tier's opponents. If you can make the games a coin flip, and then take more coin flips, you can clear a tier even with an overall poor winrate. I do think this aspect of StarSkirmish is worth addressing in the future. Astra's proxy gates beat the A-tier bots, and it moved up to StarSkirmish's S-tier (Stardust and PurpleWave, though really I'd consider BananaBrain completely our peer in ability). A brief aside about bot tournaments: The AIIDE conference and historically the COG tournaments had every bot play hundreds of games against one another, and ranked then on overall winrate. I have never liked this because it does not resemble human StarCraft (where players play a best-of-N series), unbalances the races (Zerg vs. Zerg is more volatile and winrates should be closer to 50%), and creates some twisted incentives. One of those incentives is ensuring you beat weaker opponents 100% of the time instead of merely 95%, which means defaulting to reliable-but-weaker strategies to beat weaker opponents, and having a large bag of available strategies to find something which exploits trickier opponents. And in this iteration of StarSkirmish, where benchmark bots are mostly unable to store data between games, Stardust ran one basic conservative strategy almost every game. This weakens Stardust, but meant that Astra's proxy gates were unlikely to win. So Astra had to do something very uncomfortable for a coding agent: Go back to the drawing board and try something completely different. Astra completely switched gears, landing on a strategy that PurpleWave also likes to use against Stardust: Use Dark Templar to secure an expansion. This is particularly effective against Stardust, who does not use Shuttles + Reavers, and thus does not gain utility from the Robotics Facility it requires to make Observers. From there Astra progressed slowly, but discovered a shared weakness of PurpleWave and Stardust: Both of us are designed to dodge Psionic Storms, but that feature was broken in both our bots simultaneously (for unrelated reasons!). So Astra's Dark Templar opening transitioned smoothly into a two-base timing attack with Dragoons and High Templar. It moved out as soon as it had three storms banked up. By this time Stardust or PurpleWave had usually contained the passive Astra an > — [dgant on hackernews](https://news.ycombinator.com/item?id=50122527) > > Plus, performance is just going to depend on how closely the models have memorized the best open-source Starcraft bots and can reproduce them. If that's the case then these models are failing extremely hard. 7% win rate against human written AIs that they should be copying from memory? A copy paste command should do 50%! > — [boxed on hackernews](https://news.ycombinator.com/item?id=50121519) > Also: given that the human written bots are almost certainly in the training data, the winning rate against human-written bot seems to be not well-calibrated. You would expect at least they are 50% or better, given that they can "memorize" the best human-written bot. Worth digging into the protocol and see what went wrong. > — [liuliu on hackernews](https://news.ycombinator.com/item?id=50122389) > > ... runs each model in its lab's 1st party CLI harness at the highest reasoning setting it offers. Anthropic (Claude models): Claude Code OpenAI (GPT models): Codex Google (Gemini models): Antigravity CLI Meta (Muse models): Muse Code > The CLIs' web search and web fetch tools are turned off, and the agent has no other internet access. The CLI is installed before the clock starts, and the hour counts from when the agent starts. There is no submit tool: the harness collects the bot when the time runs out. One thing we're learning is that harness is HUGE. The difference in performance for the same underlying model can change dramatically across: 1. harnesses 2. allowed tools (no web search/fetch - how does the model refer to documentation?) 3. thinking levels > — [djoldman on hackernews](https://news.ycombinator.com/item?id=50121985) > Wow this is interesting to me from another angle. A friend of mine created a benchmark to evaluate LLMs on 9x9 Go (using KataGo opponents as Elo anchors) [1]. The relative rankings of this (StarSkirmish Bench v0.2) benchmark (GPT-6 Astra > GPT-6.1 Sol > Gemini 4 Argon > Claude Opus 5.5 > Claude Fable 5.1 > Claude Sonnet 5.5) matches his GoBench rankings exactly, even though they are different problems at surface level. [1] https://rolandgao.com/blog/gobench/ > — [GodelNumbering on hackernews · 3 comments](https://news.ycombinator.com/item?id=50119857) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=50118688) · 157 points · 77 comments ## Similar posts on daily.dev - [When GPT-6 Astra started losing at StarCraft, it just stole the winning bot instead of playing fair](https://daily.dev/posts/when-gpt-6-astra-started-losing-at-starcraft-it-just-stole-the-winning-bot-instead-of-playing-fair-4opiwgett) · XDA Developers · 2 upvotes · 0 comments - [LLM Skirmish](https://daily.dev/posts/llm-skirmish-kyqo4hp5y) · Hacker News · 1 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#ai-coding](https://daily.dev/tags/ai-coding), [#claude](https://daily.dev/tags/claude), [#google-gemini](https://daily.dev/tags/google-gemini), [#gpt](https://daily.dev/tags/gpt) [View this post on daily.dev](https://daily.dev/posts/show-hn-gemini-4-argon-plays-starcraft-1dpc339s0) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Show HN: Gemini 4 Argon plays StarCraft","url":"https://daily.dev/posts/show-hn-gemini-4-argon-plays-starcraft-1dpc339s0","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/show-hn-gemini-4-argon-plays-starcraft-1dpc339s0"},"datePublished":"2026-10-16T15:47:00.225Z","dateModified":"2026-10-16T21:21:49.012Z","description":"StarSkirmish Bench v0.2 measures how well flagship LLMs (GPT-6 Astra, GPT-6.1 Sol, Gemini 4 Argon, Claude Opus 5.5, Claude Fable 5.1, Muse Wonder Full, and...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/48c78e1350c1b69b6686ffecc0c84295?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/48c78e1350c1b69b6686ffecc0c84295?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/show-hn-gemini-4-argon-plays-starcraft-1dpc339s0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,ai-coding,claude,google-gemini,gpt","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Show HN: Gemini 4 Argon plays StarCraft"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/show-hn-gemini-4-argon-plays-starcraft-1dpc339s0#faq","mainEntity":[{"@type":"Question","name":"Which LLM performs best at writing a StarCraft bot in one hour on StarSkirmish Bench v0.2","acceptedAnswer":{"@type":"Answer","text":"GPT-6 Astra scores highest on StarSkirmish Bench v0.2, a clear step ahead of GPT-6.1 Sol in second place. Gemini 4 Argon, Claude Opus 5.5, and Claude Fable 5.1 are functionally tied behind them, scoring between 37 and 39. GPT-6.1 Sol offers the best value, scoring second at roughly a tenth of GPT-6 Astra's API cost per one-hour run. Comparing flagship model performance and cost is easier when tracking benchmarks like this on daily.dev."}},{"@type":"Question","name":"How well do the best LLM-written StarCraft bots perform against top human-written bots","acceptedAnswer":{"@type":"Answer","text":"Even the best model, GPT-6 Astra, wins only 7% of its games against the strongest human-written bots Stardust, PurpleWave, and BananaBrain. This shows the benchmark is far from saturated despite rapid progress among models, with real tournament conditions against these bots being significantly harder than the three two-player maps used in testing. Developers tracking how close AI coding agents get to expert-level output can follow benchmarks like this on daily.dev."}},{"@type":"Question","name":"What is the strongest predictor of an LLM's score on StarSkirmish Bench","acceptedAnswer":{"@type":"Answer","text":"The strongest predictor of a model's score is how much it practices against the seven strongest bots on the roster during its one-hour run. Top performers like GPT-6 Astra, GPT-6.1 Sol, and Gemini 4 Argon start playing these strong bots early even while losing, whereas weaker performers like Muse Wonder Full and Claude Haiku 5.5 rarely or never do. Those evaluating agentic coding strategies can keep up with findings like this via daily.dev."}}]} ```