A benchmark comparing four security tools (Neo, Claude Code, Snyk, Invicti) against three AI-generated full-stack apps (~30,000 lines of code). Across the apps, 74 exploitable vulnerabilities were confirmed, including 21 Critical/High issues. Neo, which combines code review with runtime testing, found 66 verified findings with fewer false positives, including 24 unique vulnerabilities no other tool caught. Claude Code found 41 verified issues but produced 24 more false positives. Traditional scanners (Snyk, Invicti) missed all Critical/High issues because the serious flaws were business logic and authorization failures rather than classic signature-based bugs. The key takeaway: LLM-based code review is a hypothesis engine that can't confirm exploitability without testing a running system, and runtime validation is the critical gap to close.
