Snyk built a benchmark (Golden Tests, ~150 real vulnerable code samples across Python, JavaScript, and Java) to score whether AI-generated fixes are both secure and functional, requiring first-try passes on hidden security and functionality unit tests. Out-of-the-box frontier models (Gemini 3.1 Pro, Claude Sonnet 4.6, Claude Opus 4.6) clustered tightly at 72-75% fix rates. Adding Snyk Intelligence, a dynamic few-shot prompting layer drawing on 35,000+ expert-written fixes, lifted Claude Opus 4.6 from 74.6% to 85.4% (a 14.48% relative improvement), with the largest gains where models were weakest (Python fixes jumped from 64% to 88%). The piece concludes that security context, not raw model scale, is the lever for secure-and-functional remediation, and mentions production Agent Fix now pairs this intelligence layer with Claude Opus 4.7.
Table of contents
SummaryIntroductionHow we measure: the Golden Test benchmarkHow does the Snyk Agent Fix benchmark compare to prior benchmarks?ResultsWhat the numbers meanUp next: Snyk VulnBench and vulnerability detection with coding agentsLimitationsWhat's nextAppendix: methodology and aggregationSee Snyk in actionQuestions this post answers
How much does Snyk Intelligence improve Claude Opus 4.6's vulnerability fix rate compared to using it alone?
Snyk Intelligence raises Claude Opus 4.6's secure-and-functional fix rate from 74.6% to 85.4%, a 10.8 point gain equal to 14.48% more samples fixed. The improvement comes from dynamic few-shot prompting that injects expert-written fixes from a database of over 35,000 vulnerabilities at fix time, not from a larger or different model. Track how context-augmented AI remediation tools compare on real fix-rate benchmarks via daily.dev.
Why do frontier LLMs like Gemini 3.1 Pro and Claude Opus 4.6 struggle to produce fixes that are both secure and functional?
Out-of-the-box frontier models cluster tightly between 72.4% and 74.6% on fixes that must pass both a security test and a functional test on the first try without seeing either test, showing that raw model capability has plateaued for this task. The models can write plausible code but don't reliably know what a secure fix looks like for a specific vulnerability class without added context. Developers evaluating AI coding assistants for security-sensitive fixes can follow benchmark results like this on daily.dev.
How is a benchmark for secure and functional AI code fixes designed to avoid the model gaming the tests?
Each sample in Snyk's Golden Test set pairs a failing security unit test with a passing functional unit test, and a fix only counts if both tests pass on the first attempt while the model never sees either test. The ~150 samples span Python, JavaScript, and Java, each containing exactly one confirmed real vulnerability found by Snyk Code and verified by a human security expert. Anyone assessing AI code-fixing tools can follow rigorous benchmark methodologies like this on daily.dev.