--- title: We Automated Over 15,000 Security Reviews Without... description: An engineering team at Trendyol built an AI-assisted pipeline to automate security vulnerability verification, handling over 15,000 review tasks in four... canonical: https://daily.dev/posts/we-automated-over-15-000-security-reviews-without-letting-ai-decide-everything-oa5q3cg8k twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: We Automated Over 15,000 Security Reviews Without Letting AI Decide Everything | daily.dev og:description: An engineering team at Trendyol built an AI-assisted pipeline to automate security vulnerability verification, handling over 15,000 review tasks in four... og:url: https://daily.dev/posts/we-automated-over-15-000-security-reviews-without-letting-ai-decide-everything-oa5q3cg8k og:image: https://api.daily.dev/og/posts/oa5Q3CG8k.png og:image:alt: We Automated Over 15,000 Security Reviews Without Letting AI Decide Everything og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # We Automated Over 15,000 Security Reviews Without Letting AI Decide Everything **[Trendyol Tech](https://daily.dev/sources/trendyoltech)** · 25 min read · 15 upvotes · 1 comments ## Summary An engineering team at Trendyol built an AI-assisted pipeline to automate security vulnerability verification, handling over 15,000 review tasks in four months. The system uses a deterministic Go 'Gatekeeper' service to resolve clear-cut cases (deleted branches, expired policies) before invoking any LLM, routing only ambiguous cases to specialized AI sub-agents (Secret, SAST, Manual, AI) coordinated by an Orchestrator. A three-way verdict system (Resolved/Not Resolved/INCONCLUSIVE) lets the system abstain rather than force a binary decision when evidence is insufficient. Agreement with human reviewers rose from about 60% at shadow launch to 97.86% after deterministic filtering and prompt refinements, while automating 87.02% of closures and keeping LLM spend near $0.09 per task ($1,391.32 total). A proof-of-concept 'Sentinel' agent analyzes failure traces to propose targeted prompt fixes, though it remains human-reviewed rather than self-modifying. The team estimates the system saved roughly 161 engineer-days of review effort. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do you decide when it's safe to let AI automatically verify a security vulnerability fix instead of a human? Automation should be earned through evidence, not granted by default: start in an analyze-only shadow mode comparing AI verdicts against human decisions, then promote each scanner path independently once agreement is consistently high. One system moved from about 60% AI-human agreement at shadow launch to 97.86% after removing deterministic cases with pre-flight rules and refining prompts based on mismatch analysis, before enabling autonomous auto-retest per scanner type. _daily.dev surfaces engineering writeups like this for teams designing staged rollouts of AI decision automation._ ### Why would you add a third 'inconclusive' verdict instead of just fixed or not fixed in an automated verification system? Because binary classification becomes dangerous when evidence is incomplete: a timeout, authentication failure, or missing trace doesn't prove a vulnerability is fixed or still active. In one system, 344 of 15,344 AI-analyzed tasks (2.24%) ended in INCONCLUSIVE and were escalated to human review rather than force-converted into a Resolved or Not Resolved verdict, keeping the abstention rate itself as a measurable signal for improvement. _developers designing safe AI automation for high-stakes decisions can track approaches like this on daily.dev._ ### What's the cost-effective way to reduce LLM spend when building an AI verification pipeline for security findings? Filter out deterministic cases with regular code before any LLM is invoked, since a service that can resolve a case conclusively (like a deleted branch, expired policy, or a Trivy container scan match) costs zero in LLM spend. Routing everything through a model instead maximizes cost, latency, and error surface. This approach kept average spend to about $0.09 per AI-analyzed task across 15,344 tasks, totaling $1,391.32. _daily.dev helps engineers researching cost-efficient LLM pipeline design keep up with real-world case studies like this._ ## Community discussion Top comments from developers on daily.dev. **@ristotoldsep** · 2 upvotes > The `INCONCLUSIVE` verdict is the smartest part. Security doesn’t need fake certainty. ## Similar posts on daily.dev - [Automating Security Champion Workflows with AI Agents at Trendyol](https://daily.dev/posts/automating-security-champion-workflows-with-ai-agents-at-trendyol-1tke6zjua) · Trendyol Tech · 0 upvotes · 0 comments - [Orchestrating AI Code Review at scale](https://daily.dev/posts/orchestrating-ai-code-review-at-scale-re4wh2np4) · Cloudflare · 0 upvotes · 0 comments - [Using AI agents to secure Google infrastructure](https://daily.dev/posts/using-ai-agents-to-secure-google-infrastructure-n2sosodh7) · Google Cloud · 0 upvotes · 0 comments - [Agentic Code Review](https://daily.dev/posts/agentic-code-review-0ue8fcsa1) · Addy Osmani · 146 upvotes · 14 comments --- Tags: [#security](https://daily.dev/tags/security), [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents), [#golang](https://daily.dev/tags/golang), [#appsec](https://daily.dev/tags/appsec) [View this post on daily.dev](https://daily.dev/posts/we-automated-over-15-000-security-reviews-without-letting-ai-decide-everything-oa5q3cg8k) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"We Automated Over 15,000 Security Reviews Without Letting AI Decide Everything","url":"https://daily.dev/posts/we-automated-over-15-000-security-reviews-without-letting-ai-decide-everything-oa5q3cg8k","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/we-automated-over-15-000-security-reviews-without-letting-ai-decide-everything-oa5q3cg8k"},"datePublished":"2026-09-16T19:32:43.736Z","dateModified":"2026-09-16T20:55:22.074Z","description":"An engineering team at Trendyol built an AI-assisted pipeline to automate security vulnerability verification, handling over 15,000 review tasks in four...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/37066b83ff4c138c2ed8fc008b3cbd41?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/37066b83ff4c138c2ed8fc008b3cbd41?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Trendyol Tech","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Trendyol Tech","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/67480ca531174b03952be35abd6572a9","url":"https://daily.dev/sources/trendyoltech"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/we-automated-over-15-000-security-reviews-without-letting-ai-decide-everything-oa5q3cg8k","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":15},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"security,llm,ai-agents,golang,appsec","timeRequired":"PT25M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Trendyol Tech","item":"https://daily.dev/sources/trendyoltech"},{"@type":"ListItem","position":3,"name":"We Automated Over 15,000 Security Reviews Without Letting AI Decide Everything"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/we-automated-over-15-000-security-reviews-without-letting-ai-decide-everything-oa5q3cg8k","comment":[{"@type":"Comment","text":"The INCONCLUSIVE verdict is the smartest part. Security doesn’t need fake certainty.","datePublished":"2026-09-22T07:31:33.011Z","url":"https://daily.dev/posts/oa5Q3CG8k#c-rUsgBz6Xk","author":{"@type":"Person","name":"Risto Tõldsep","url":"https://daily.dev/ristotoldsep","image":"https://lh3.googleusercontent.com/a/ACg8ocLDWc6mZn0JwNmXXw6WY0L_HJ6pegzRttooC5VgtXESHSj3MNYx=s96-c"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/we-automated-over-15-000-security-reviews-without-letting-ai-decide-everything-oa5q3cg8k#faq","mainEntity":[{"@type":"Question","name":"How do you decide when it's safe to let AI automatically verify a security vulnerability fix instead of a human?","acceptedAnswer":{"@type":"Answer","text":"Automation should be earned through evidence, not granted by default: start in an analyze-only shadow mode comparing AI verdicts against human decisions, then promote each scanner path independently once agreement is consistently high. One system moved from about 60% AI-human agreement at shadow launch to 97.86% after removing deterministic cases with pre-flight rules and refining prompts based on mismatch analysis, before enabling autonomous auto-retest per scanner type. daily.dev surfaces engineering writeups like this for teams designing staged rollouts of AI decision automation."}},{"@type":"Question","name":"Why would you add a third 'inconclusive' verdict instead of just fixed or not fixed in an automated verification system?","acceptedAnswer":{"@type":"Answer","text":"Because binary classification becomes dangerous when evidence is incomplete: a timeout, authentication failure, or missing trace doesn't prove a vulnerability is fixed or still active. In one system, 344 of 15,344 AI-analyzed tasks (2.24%) ended in INCONCLUSIVE and were escalated to human review rather than force-converted into a Resolved or Not Resolved verdict, keeping the abstention rate itself as a measurable signal for improvement. developers designing safe AI automation for high-stakes decisions can track approaches like this on daily.dev."}},{"@type":"Question","name":"What's the cost-effective way to reduce LLM spend when building an AI verification pipeline for security findings?","acceptedAnswer":{"@type":"Answer","text":"Filter out deterministic cases with regular code before any LLM is invoked, since a service that can resolve a case conclusively (like a deleted branch, expired policy, or a Trivy container scan match) costs zero in LLM spend. Routing everything through a model instead maximizes cost, latency, and error surface. This approach kept average spend to about $0.09 per AI-analyzed task across 15,344 tasks, totaling $1,391.32. daily.dev helps engineers researching cost-efficient LLM pipeline design keep up with real-world case studies like this."}}]} ```