A puzzle-game benchmark called Baba Is Bench was rerun against a fresh wave of LLM releases: Gemini 3.7 Flash, Grok 4.6, DeepSeek V4 Pro 0813, Qwen3.8 Max/27B, and GLM-5.3. Gemini 3.7 Flash, Grok 4.6, and DeepSeek V4 Pro 0813 each showed large jumps in pass rate over their predecessors while costing 3-20x less, with DeepSeek V4 Pro 0813 becoming the first open-weight model to reach the top tier on the harder 'Lake' stage. GLM-5.3 barely improved over GLM-5.2, and Qwen3.8 models showed mixed or worse results than prior versions, with Qwen3.8 27B reportedly overthinking rather than exploring the board. Cost and speed comparisons across models are broken down using turns, output tokens, wall-clock time, and total dollar cost, alongside notes on harness choice (Harbor's Terminus-2 vs Claude Code vs vendor-specific harnesses like Grok Build and DeepSeek Harness) affecting results.
Questions this post answers
How does DeepSeek V4 Pro 0813 compare to the previous DeepSeek V4 Pro on the Baba Is Bench puzzle benchmark?
DeepSeek V4 Pro 0813 solved 7 out of 8 intro levels compared to only 3 out of 8 for the previous DeepSeek V4 Pro, while costing about 4x less. On the harder Lake stage it became the first open-weight model to reach the same top league as frontier closed models, and it ended up the cheapest model tested overall. Track how open-weight models like DeepSeek close the gap with frontier releases via daily.dev.
How much cheaper is Gemini 3.7 Flash compared to Gemini 3.6 Flash on agentic benchmark tasks?
Gemini 3.7 Flash is over 20x cheaper than Gemini 3.6 Flash while now solving every trial in the intro stage of the Baba Is Bench benchmark, versus 88% for the previous version. On the harder Lake stage it is also markedly faster per problem solved, with total cost roughly half that of the previous cheapest model, GPT-5.6 Sol. Compare model pricing and performance shifts like this on daily.dev before picking an API for agentic work.
Why did Qwen3.8 27B perform worse than expected on agentic puzzle-solving tasks despite scoring well on Artificial Analysis benchmarks?
Qwen3.8 27B tended to try solving problems in one go without actually exploring the board, defaulting to overthinking rather than balancing thinking with testing. This matches independent reports of the same model overthinking simple problems, and it produced worse Baba Is Bench results than the prior Qwen3.6 27B, even though it scores 52 on Artificial Analysis. See how benchmark scores translate to real agentic performance before choosing a model, on daily.dev.