We burned 11.7bn tokens to find the best cyber AI model
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Aikido Security benchmarked 10 AI models (including new entrants GLM-5.3, DeepSeek V4 Pro/Flash, Qwen3.8-Max, Kimi K3, and Grok 4.6) against 32 freshly disclosed CVEs, running each model three times for 96 total runs. DeepSeek V4 Pro 0813 found the most vulnerabilities overall, reaching 28 of 32 when pooling three runs, while costing roughly $295 versus $450-590 for a single pass of closed frontier models like Opus 5, Grok 4.6, or Sol. Grok was the most consistent model, replicating 21 of its findings across all three runs. Open-weight models (DeepSeek, GLM-5.3, Qwen, Kimi) now match or beat public closed frontier models on pooled recall, though they generate more false positives requiring downstream triage. The piece argues repetition and pooling of multiple cheap runs can outperform a single expensive run, but stresses that open models still require careful harness engineering to be production-ready.
Table of contents
How the benchmark worksMaking model output variance work for usDeepSeek had the highest pooled recall. Grok was the most consistentInside the investigation tracesGLM closes the frontier gapOpen weights just beat the public frontierSo which model would we hire?Questions this post answers
Which AI model found the most vulnerabilities in a CVE rediscovery benchmark, DeepSeek or Opus 5?
DeepSeek V4 Pro 0813 found the most, reaching 28 of 32 vulnerabilities when pooling three runs, compared to 26 of 32 for both Opus 5 and Grok 4.6. DeepSeek's first pass alone found only 17, showing pooling multiple runs significantly improved recall. Three DeepSeek Pro runs cost about $295, versus $450-590 for a single Opus 5, Grok 4.6, or Sol pass. Teams weighing cheap pooled AI scans against premium single-pass models can track these tradeoffs on daily.dev.
Do open-weight AI models like DeepSeek and GLM-5.3 outperform closed frontier models for security vulnerability discovery?
Yes, in a benchmark of 32 fresh CVEs across 10 models, open-weight models including DeepSeek V4 Pro, GLM-5.3, Qwen3.8-Max, and Kimi K3 matched or beat closed frontier models like Opus 5, Grok 4.6, and Sol on pooled vulnerability recall. However, the open models produced more false-positive findings requiring extra triage, and still need careful harness engineering rather than a drop-in replacement. Developers choosing between open and closed AI models for security tooling can follow these comparisons on daily.dev.
How much does it cost to run DeepSeek V4 Flash three times for vulnerability scanning versus one Grok 4.6 pass?
Three DeepSeek V4 Flash 0731 runs cost about $108 total and found 24 of 32 vulnerabilities, matching Grok 4.6's best single pass for less than a quarter of the price. This demonstrates that repeating a cheaper model across multiple runs can rival the coverage of one expensive frontier model pass, though it shifts more false-positive triage burden downstream. daily.dev helps engineers weighing cost versus coverage keep up with AI model cost benchmarks like this.