<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/we-burned-11-7bn-tokens-to-find-the-best-cyber-ai-model-tbgc2dpsf" -->

---
title: We burned 11.7bn tokens to find the best cyber AI model
description: Aikido Security benchmarked 10 AI models (including new entrants GLM-5.3, DeepSeek V4 Pro/Flash, Qwen3.8-Max, Kimi K3, and Grok 4.6) against 32 freshly...
canonical: https://daily.dev/posts/we-burned-11-7bn-tokens-to-find-the-best-cyber-ai-model-tbgc2dpsf
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: We burned 11.7bn tokens to find the best cyber AI model | daily.dev
og:description: Aikido Security benchmarked 10 AI models (including new entrants GLM-5.3, DeepSeek V4 Pro/Flash, Qwen3.8-Max, Kimi K3, and Grok 4.6) against 32 freshly...
og:url: https://daily.dev/posts/we-burned-11-7bn-tokens-to-find-the-best-cyber-ai-model-tbgc2dpsf
og:image: https://api.daily.dev/og/posts/tBGc2dpsF.png
og:image:alt: We burned 11.7bn tokens to find the best cyber AI model
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# We burned 11.7bn tokens to find the best cyber AI model

**[Aikido Security](https://daily.dev/sources/aikido-security)** · 10 min read · 0 upvotes · 0 comments

## Summary

Aikido Security benchmarked 10 AI models (including new entrants GLM-5.3, DeepSeek V4 Pro/Flash, Qwen3.8-Max, Kimi K3, and Grok 4.6) against 32 freshly disclosed CVEs, running each model three times for 96 total runs. DeepSeek V4 Pro 0813 found the most vulnerabilities overall, reaching 28 of 32 when pooling three runs, while costing roughly $295 versus $450-590 for a single pass of closed frontier models like Opus 5, Grok 4.6, or Sol. Grok was the most consistent model, replicating 21 of its findings across all three runs. Open-weight models (DeepSeek, GLM-5.3, Qwen, Kimi) now match or beat public closed frontier models on pooled recall, though they generate more false positives requiring downstream triage. The piece argues repetition and pooling of multiple cheap runs can outperform a single expensive run, but stresses that open models still require careful harness engineering to be production-ready.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://www.aikido.dev/blog/ai-model-benchmarks-aug-21-2026>

## Questions this post answers

### Which AI model found the most vulnerabilities in a CVE rediscovery benchmark, DeepSeek or Opus?

DeepSeek V4 Pro 0813 found the most vulnerabilities, reaching 28 of 32 fresh CVEs when pooling three runs, ahead of Opus 5, Grok 4.6, and Sol, which each found 25-26 of 32. DeepSeek's first pass alone only found 17, showing pooled runs mattered more than raw single-pass performance for this model.

_Comparing AI security tools before committing budget is easier with benchmark write-ups surfaced on daily.dev._

### How much does it cost to run three passes of DeepSeek V4 Pro versus one pass of Opus 5 for vulnerability scanning?

Three runs of DeepSeek V4 Pro 0813 cost about $295 total and found 28 of 32 vulnerabilities, while a single pass of Opus 5, Grok 4.6, or Sol cost roughly $450-$590 and found fewer vulnerabilities than the pooled DeepSeek runs. Three DeepSeek Flash runs cost just $108 and matched Grok's best single pass at 24 vulnerabilities.

_Weighing cost against coverage for AI security tooling gets easier tracking findings like this on daily.dev._

### Why did Qwen3.8-Max waste investigation turns during a vulnerability discovery benchmark?

Qwen repeatedly recognized codebases and spent turns recalling older, already-patched CVEs from its training data in 95 of 96 benchmark runs, wasting investigation budget before returning to actual code reasoning. Despite this quirk, it still climbed the leaderboard, matching Opus and Grok on pooled recall at pass@3, just behind DeepSeek's lead.

_Understanding quirks like training-data recall helps engineers pick the right model, a call daily.dev helps inform._

## Similar posts on daily.dev

- [Benchmarking 13 AI Models on Known CVE Detection](https://daily.dev/posts/benchmarking-13-ai-models-on-known-cve-detection-n1jxpyasp) · Aikido Security · 2 upvotes · 1 comments
- [We Tested Grok, Opus, Sonnet, GPT, and Gemini in Kilo Code Reviewer](https://daily.dev/posts/we-tested-grok-opus-sonnet-gpt-and-gemini-in-kilo-code-reviewer-yjkkqinsy) · Kilo Blog · 8 upvotes · 1 comments
- [Gemini 3.7 Flash, Grok 4.6, GLM-5.3 and DeepSeek V4 Pro joined the frontier](https://daily.dev/posts/gemini-3-7-flash-grok-4-6-glm-5-3-and-deepseek-v4-pro-joined-the-frontier-pfyczsmpb) · Quesma · 0 upvotes · 0 comments

---

Tags: [#ai](https://daily.dev/tags/ai), [#vulnerability](https://daily.dev/tags/vulnerability), [#appsec](https://daily.dev/tags/appsec), [#deepseek](https://daily.dev/tags/deepseek)

[View this post on daily.dev](https://daily.dev/posts/we-burned-11-7bn-tokens-to-find-the-best-cyber-ai-model-tbgc2dpsf)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"TechArticle","headline":"We burned 11.7bn tokens to find the best cyber AI model","url":"https://daily.dev/posts/we-burned-11-7bn-tokens-to-find-the-best-cyber-ai-model-tbgc2dpsf","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/we-burned-11-7bn-tokens-to-find-the-best-cyber-ai-model-tbgc2dpsf"},"datePublished":"2026-08-21T09:19:37.398Z","dateModified":"2026-09-14T08:50:27.849Z","description":"Aikido Security benchmarked 10 AI models (including new entrants GLM-5.3, DeepSeek V4 Pro/Flash, Qwen3.8-Max, Kimi K3, and Grok 4.6) against 32 freshly...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/a888abfd862b62f23f1f0457ed8ab2fc?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/a888abfd862b62f23f1f0457ed8ab2fc?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Aikido Security","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Aikido Security","logo":"https://media.daily.dev/image/upload/s--vJDqDYv_--/f_auto,q_auto/v1774959928/logos/aikido-security?_a=BAMAMiWQ0","url":"https://daily.dev/sources/aikido-security"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/we-burned-11-7bn-tokens-to-find-the-best-cyber-ai-model-tbgc2dpsf","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,vulnerability,appsec,deepseek","timeRequired":"PT10M"}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Aikido Security","item":"https://daily.dev/sources/aikido-security"},{"@type":"ListItem","position":3,"name":"We burned 11.7bn tokens to find the best cyber AI model"}]}
{"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/we-burned-11-7bn-tokens-to-find-the-best-cyber-ai-model-tbgc2dpsf#faq","mainEntity":[{"@type":"Question","name":"Which AI model found the most vulnerabilities in a CVE rediscovery benchmark, DeepSeek or Opus?","acceptedAnswer":{"@type":"Answer","text":"DeepSeek V4 Pro 0813 found the most vulnerabilities, reaching 28 of 32 fresh CVEs when pooling three runs, ahead of Opus 5, Grok 4.6, and Sol, which each found 25-26 of 32. DeepSeek's first pass alone only found 17, showing pooled runs mattered more than raw single-pass performance for this model. Comparing AI security tools before committing budget is easier with benchmark write-ups surfaced on daily.dev."}},{"@type":"Question","name":"How much does it cost to run three passes of DeepSeek V4 Pro versus one pass of Opus 5 for vulnerability scanning?","acceptedAnswer":{"@type":"Answer","text":"Three runs of DeepSeek V4 Pro 0813 cost about $295 total and found 28 of 32 vulnerabilities, while a single pass of Opus 5, Grok 4.6, or Sol cost roughly $450-$590 and found fewer vulnerabilities than the pooled DeepSeek runs. Three DeepSeek Flash runs cost just $108 and matched Grok's best single pass at 24 vulnerabilities. Weighing cost against coverage for AI security tooling gets easier tracking findings like this on daily.dev."}},{"@type":"Question","name":"Why did Qwen3.8-Max waste investigation turns during a vulnerability discovery benchmark?","acceptedAnswer":{"@type":"Answer","text":"Qwen repeatedly recognized codebases and spent turns recalling older, already-patched CVEs from its training data in 95 of 96 benchmark runs, wasting investigation budget before returning to actual code reasoning. Despite this quirk, it still climbed the leaderboard, matching Opus and Grok on pooled recall at pass@3, just behind DeepSeek's lead. Understanding quirks like training-data recall helps engineers pick the right model, a call daily.dev helps inform."}}]}
```