--- title: Teaching AI to Find Real Vulnerabilities — David... description: David Brumley (CMU professor and Bugcrowd Chief AI Officer) presents a framework for training LLMs to find and exploit real vulnerabilities using reinforcement... canonical: https://daily.dev/posts/teaching-ai-to-find-real-vulnerabilities-david-brumley-bugcrowd-vbz44q16c twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Teaching AI to Find Real Vulnerabilities — David Brumley, Bugcrowd | daily.dev og:description: David Brumley (CMU professor and Bugcrowd Chief AI Officer) presents a framework for training LLMs to find and exploit real vulnerabilities using reinforcement... og:url: https://daily.dev/posts/teaching-ai-to-find-real-vulnerabilities-david-brumley-bugcrowd-vbz44q16c og:image: https://api.daily.dev/og/posts/VBZ44Q16C.png og:image:alt: Teaching AI to Find Real Vulnerabilities — David Brumley, Bugcrowd og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Teaching AI to Find Real Vulnerabilities — David Brumley, Bugcrowd **[AI Engineer](https://daily.dev/sources/aidotengineer)** · 27 min read · 1 upvotes · 2 comments ## Summary David Brumley (CMU professor and Bugcrowd Chief AI Officer) presents a framework for training LLMs to find and exploit real vulnerabilities using reinforcement learning. He identifies critical flaws in existing cybersecurity benchmarks — primarily that programs with multiple vulnerabilities cause models to reward-hack by repeatedly finding the easiest bug. His solution is an 'audit task' approach using precision/recall scoring across all discovered vulnerabilities, with deterministic grading oracles instead of LLM-as-judge. He then shares results from testing frontier models (GPT, Mythos/Claude, Gemini) against 41 real V8 (Chrome) vulnerabilities, finding that top models achieved full sandbox escape and arbitrary code execution 68-73% of the time — including novel exploit paths not previously known publicly. The benchmark environments are available at exploitbench.ai as Docker images with MCP interfaces. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Community discussion Top comments from developers on daily.dev. **@agustinbarrientos** · 0 upvotes > An LLM grading whether another LLM got code execution sounds far too optimistic. **@trevorsuna** · 0 upvotes > Precision and recall across every vulnerability feels much closer to a real audit than rewarding the first easy bug. Deterministic grading is especially important here; otherwise the judge becomes another attack surface for reward hacking. ## Similar posts on daily.dev - [Using LLMs to Find and Prioritize Vulnerabilities Is No Easy Task](https://daily.dev/posts/using-llms-to-find-and-prioritize-vulnerabilities-is-no-easy-task-dtmgdglfx) · Dark Reading · 5 upvotes · 1 comments --- Tags: [#security](https://daily.dev/tags/security), [#llm](https://daily.dev/tags/llm), [#reinforcement-learning](https://daily.dev/tags/reinforcement-learning), [#v8](https://daily.dev/tags/v8) [View this post on daily.dev](https://daily.dev/posts/teaching-ai-to-find-real-vulnerabilities-david-brumley-bugcrowd-vbz44q16c) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Teaching AI to Find Real Vulnerabilities — David Brumley, Bugcrowd","url":"https://daily.dev/posts/teaching-ai-to-find-real-vulnerabilities-david-brumley-bugcrowd-vbz44q16c","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/teaching-ai-to-find-real-vulnerabilities-david-brumley-bugcrowd-vbz44q16c"},"datePublished":"2026-08-01T00:32:04.415Z","dateModified":"2026-08-01T06:18:01.577Z","description":"David Brumley (CMU professor and Bugcrowd Chief AI Officer) presents a framework for training LLMs to find and exploit real vulnerabilities using reinforcement...","image":"https://i.ytimg.com/vi/ZFxh7sqbUZo/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/ZFxh7sqbUZo/sddefault.jpg","isAccessibleForFree":true,"articleSection":"AI Engineer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"AI Engineer","logo":"https://media.daily.dev/image/upload/s--u5PucxNT--/f_auto/v1724338940/logos/aidotengineer","url":"https://daily.dev/sources/aidotengineer"},"commentCount":2,"discussionUrl":"https://daily.dev/posts/teaching-ai-to-find-real-vulnerabilities-david-brumley-bugcrowd-vbz44q16c","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":2}],"keywords":"security,llm,reinforcement-learning,v8","timeRequired":"PT27M","video":{"@type":"VideoObject","name":"Teaching AI to Find Real Vulnerabilities — David Brumley, Bugcrowd","description":"David Brumley (CMU professor and Bugcrowd Chief AI Officer) presents a framework for training LLMs to find and exploit real vulnerabilities using reinforcement...","thumbnailUrl":"https://i.ytimg.com/vi/ZFxh7sqbUZo/sddefault.jpg","uploadDate":"2026-08-01T00:32:04.415Z","duration":"PT27M","url":"https://api.daily.dev/r/VBZ44Q16C","embedUrl":"https://www.youtube.com/embed/ZFxh7sqbUZo"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AI Engineer","item":"https://daily.dev/sources/aidotengineer"},{"@type":"ListItem","position":3,"name":"Teaching AI to Find Real Vulnerabilities — David Brumley, Bugcrowd"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/teaching-ai-to-find-real-vulnerabilities-david-brumley-bugcrowd-vbz44q16c","comment":[{"@type":"Comment","text":"An LLM grading whether another LLM got code execution sounds far too optimistic.","datePublished":"2026-08-02T19:47:00.352Z","url":"https://daily.dev/posts/VBZ44Q16C#c-LQFdyDb3T","author":{"@type":"Person","name":"Agustin Barrientos","url":"https://daily.dev/agustinbarrientos","image":"https://media.daily.dev/image/upload/s--5ayxQnqn--/f_auto/v1788281802/avatars/avatar_wQYYVe5Tbj0NJ7C7qPoa8?_a=BAMAMicg0"}},{"@type":"Comment","text":"Precision and recall across every vulnerability feels much closer to a real audit than rewarding the first easy bug. Deterministic grading is especially important here; otherwise the judge becomes another attack surface for reward hacking.","datePublished":"2026-08-01T05:58:03.536Z","url":"https://daily.dev/posts/VBZ44Q16C#c-stBk2R3QC","author":{"@type":"Person","name":"Trevor Suna","url":"https://daily.dev/trevorsuna","image":"https://media.daily.dev/image/upload/s--dZ7gXxpp--/f_auto/v1784081551/avatars/avatar_EMoP47rpuw8DNjhp6R1b6?_a=BAMAMicg0"}}]} ```