--- title: "AI code benchmarks lied to us" url: https://daily.dev/posts/ai-code-benchmarks-lied-to-us-o4e8ldyyt source_url: https://www.youtube.com/watch?v=JpSHyEIZ_bo type: video:youtube source: "Theo - t3․gg" published: 2026-05-31T08:56:11.251Z updated: 2026-06-02T22:48:21.837Z tags: ["llm", "ai-coding"] reading_time: 32 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # AI code benchmarks lied to us **[Theo - t3․gg](https://daily.dev/sources/t3dotgg)** · 32 min read · 0 upvotes · 0 comments ## Summary A deep critique of existing AI coding benchmarks, particularly SWE-bench Pro, arguing they are contaminated, poorly designed, and produce misleading results. The author introduces DeepSWE bench (by Data Curve, a company they've invested in) as a more realistic alternative that tests models on novel tasks across multiple languages with handwritten verifiers. Key findings: GPT-4.5/o3 (called GPT55/54) dramatically outperform other models, Gemini Flash performs far worse than SWE-bench Pro suggests, and Claude Opus uses 2-3x more tokens at higher cost for lower scores. The benchmark reveals a massive gap between open-weight models and frontier models that older benchmarks obscured. The author also encourages developers to build their own mini-benchmarks from real failure cases. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Why SWE-bench Verified no longer measures frontier coding capabilities](https://daily.dev/posts/why-swe-bench-verified-no-longer-measures-frontier-coding-capabilities-yitnqurmf) · Hacker News · 5 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#ai-coding](https://daily.dev/tags/ai-coding) [View this post on daily.dev](https://daily.dev/posts/ai-code-benchmarks-lied-to-us-o4e8ldyyt)