Don't Let the AI Find Your Bugs. Let It Judge Them.
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
An architecture for a Java vulnerability scanner splits work between deterministic taint-tracking rules (via Joern, a static analysis tool building Code Property Graphs) that exhaustively but noisily find candidate flows, and an LLM judge (Gemma) that reviews each small code slice to decide if it's a real vulnerability, its severity, and a fix. The author argues LLMs are unreliable at open-ended searching (nondeterministic output, position bias, hallucination) but good at judging a single small, well-defined question. Tested against the full OWASP Benchmark, the rules-only layer matches CodeQL's recall (1.00) but with more false positives (614 vs 427); adding the LLM judge cuts false alarms by 52% while losing only 2% of true positives, moving F1 from 0.72 to ~0.79, still short of CodeQL's 0.78 vs their claimed range. A related 2025 arXiv paper (LLMxCPG) uses a similar CPG-plus-LLM-judgment approach.
Table of contents
The ContextThe Problem With "Let the AI Find the Bugs"The Problem With the Opposite: "Just Use Rules"The Real Solution: A CourtroomResults So FarWhat I LearnedQuestions this post answers
Why do AI agents that scan an entire codebase for vulnerabilities perform poorly compared to traditional static analysis tools like Semgrep or CodeQL?
LLMs struggle with exhaustive, open-ended searching across a codebase because they are probabilistic, produce different results on repeated runs of the same input, exhibit position bias that causes them to skim or lose focus in long files, and can hallucinate vulnerabilities that don't exist. This makes their output impossible to benchmark reliably against deterministic tools like Semgrep or CodeQL, which always return the same results for the same input. Anyone weighing AI-based versus rule-based scanners can compare real trade-offs like this on daily.dev.
How much does adding an LLM judge on top of deterministic taint-tracking rules improve precision in vulnerability scanning?
In a test against the OWASP Benchmark's 1,478 labeled Java cases, a seven-row deterministic rule set alone achieved 0.56 precision and 1.00 recall (F1 0.72), while CodeQL achieved 0.65 precision and 1.00 recall (F1 0.78). Adding an LLM judge (Gemma) that reviews each flagged candidate raised precision to 0.67 while retaining 0.98 recall, cutting false alarms by 52% while losing only 2% of true positives, for an F1 around 0.79. Teams evaluating AI-augmented security tooling can track benchmarks like this one on daily.dev.
How fast can Joern's Code Property Graph analysis scan a large Java codebase for vulnerability candidates?
Joern, an open-source static analysis tool, built a Code Property Graph over 2,766 files in 100 seconds and then ran a full seven-row taint query library across that graph in just 9 seconds, demonstrating that deterministic discovery of vulnerability candidates is cheap and fast even run from a laptop, leaving the LLM's expensive judgment step to be applied only to the small set of flagged candidates. Engineers scoping static analysis pipelines can follow performance breakdowns like this on daily.dev.
3.8K Impressions1 Comment