--- title: "Don't Let the AI Find Your Bugs. Let It Judge Them." url: https://daily.dev/posts/don-t-let-the-ai-find-your-bugs-let-it-judge-them--fpqkjl4lo source_url: https://dev.to/alimafana/dont-let-the-ai-find-your-bugs-let-it-judge-them-5dbp type: article source: "Awesome Java Newsletter" published: 2026-08-20T20:24:37.940Z updated: 2026-08-20T20:39:17.838Z tags: ["llm", "java", "sql", "appsec"] reading_time: 11 upvotes: 2 comments: 1 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Don't Let the AI Find Your Bugs. Let It Judge Them. **[Awesome Java Newsletter](https://daily.dev/sources/java_libhunt)** · 11 min read · 2 upvotes · 1 comments ## Summary An architecture for a Java vulnerability scanner splits work between deterministic taint-tracking rules (via Joern, a static analysis tool building Code Property Graphs) that exhaustively but noisily find candidate flows, and an LLM judge (Gemma) that reviews each small code slice to decide if it's a real vulnerability, its severity, and a fix. The author argues LLMs are unreliable at open-ended searching (nondeterministic output, position bias, hallucination) but good at judging a single small, well-defined question. Tested against the full OWASP Benchmark, the rules-only layer matches CodeQL's recall (1.00) but with more false positives (614 vs 427); adding the LLM judge cuts false alarms by 52% while losing only 2% of true positives, moving F1 from 0.72 to ~0.79, still short of CodeQL's 0.78 vs their claimed range. A related 2025 arXiv paper (LLMxCPG) uses a similar CPG-plus-LLM-judgment approach. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why do AI agents that scan an entire codebase for vulnerabilities perform poorly compared to traditional static analysis tools like Semgrep or CodeQL? LLMs struggle with exhaustive, open-ended searching across a codebase because they are probabilistic, produce different results on repeated runs of the same input, exhibit position bias that causes them to skim or lose focus in long files, and can hallucinate vulnerabilities that don't exist. This makes their output impossible to benchmark reliably against deterministic tools like Semgrep or CodeQL, which always return the same results for the same input. _Anyone weighing AI-based versus rule-based scanners can compare real trade-offs like this on daily.dev._ ### How much does adding an LLM judge on top of deterministic taint-tracking rules improve precision in vulnerability scanning? In a test against the OWASP Benchmark's 1,478 labeled Java cases, a seven-row deterministic rule set alone achieved 0.56 precision and 1.00 recall (F1 0.72), while CodeQL achieved 0.65 precision and 1.00 recall (F1 0.78). Adding an LLM judge (Gemma) that reviews each flagged candidate raised precision to 0.67 while retaining 0.98 recall, cutting false alarms by 52% while losing only 2% of true positives, for an F1 around 0.79. _Teams evaluating AI-augmented security tooling can track benchmarks like this one on daily.dev._ ### How fast can Joern's Code Property Graph analysis scan a large Java codebase for vulnerability candidates? Joern, an open-source static analysis tool, built a Code Property Graph over 2,766 files in 100 seconds and then ran a full seven-row taint query library across that graph in just 9 seconds, demonstrating that deterministic discovery of vulnerability candidates is cheap and fast even run from a laptop, leaving the LLM's expensive judgment step to be applied only to the small set of flagged candidates. _Engineers scoping static analysis pipelines can follow performance breakdowns like this on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@trevorsuna** · 0 upvotes > Using deterministic analysis for recall and an LLM only for narrow classification is a sensible split. I’d also keep the rejected findings for sampling and regression tests, because a lower false-positive rate is valuable only if the judge’s misses stay visible over time. ## Similar posts on daily.dev - [LLM Security Automation Isn’t a Drop-In Scanner Yet](https://daily.dev/posts/llm-security-automation-isn-t-a-drop-in-scanner-yet-08iduwsiq) · Liran Tal · 0 upvotes · 0 comments - [Using LLMs to Find and Prioritize Vulnerabilities Is No Easy Task](https://daily.dev/posts/using-llms-to-find-and-prioritize-vulnerabilities-is-no-easy-task-dtmgdglfx) · Dark Reading · 5 upvotes · 1 comments - [Using LLMs to Find Security Bugs: A Practitioner’s Playbook](https://daily.dev/posts/using-llms-to-find-security-bugs-a-practitioner-s-playbook-llrqsqe4g) · Tech Lead Digest · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#java](https://daily.dev/tags/java), [#sql](https://daily.dev/tags/sql), [#appsec](https://daily.dev/tags/appsec) [View this post on daily.dev](https://daily.dev/posts/don-t-let-the-ai-find-your-bugs-let-it-judge-them--fpqkjl4lo)