--- title: We Had 13 Engineers Spend Three Months Finding... description: Teleport dedicated 13 engineers for three months to use frontier LLMs to find security vulnerabilities in their codebase, a process they call 'pressure... canonical: https://daily.dev/posts/we-had-13-engineers-spend-three-months-finding-vulnerabilities-with-llms-xrbmxtlki twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: We Had 13 Engineers Spend Three Months Finding Vulnerabilities with LLMs | daily.dev og:description: Teleport dedicated 13 engineers for three months to use frontier LLMs to find security vulnerabilities in their codebase, a process they call 'pressure... og:url: https://daily.dev/posts/we-had-13-engineers-spend-three-months-finding-vulnerabilities-with-llms-xrbmxtlki og:image: https://api.daily.dev/og/posts/XRBmXtlKI.png og:image:alt: We Had 13 Engineers Spend Three Months Finding Vulnerabilities with LLMs og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # We Had 13 Engineers Spend Three Months Finding Vulnerabilities with LLMs **[Teleport](https://daily.dev/sources/teleport)** · 5 min read · 5 upvotes · 8 comments ## Summary Teleport dedicated 13 engineers for three months to use frontier LLMs to find security vulnerabilities in their codebase, a process they call 'pressure washing.' They found that complicated multi-agent harnesses (like their own tool, Conclave) underperformed simple, targeted prompts guided by experienced human vulnerability researchers. One effective prompt framed the task as a CTF challenge focused on a specific file. The effort surfaced dozens of security bugs, nearly doubling the high-severity vulnerabilities found compared to the previous two years combined, resulting in larger-than-usual security releases including version 18.10.7. The key takeaway is that LLMs currently amplify expert human judgment rather than replace it, and going forward Teleport plans to keep working with external researchers like Calif and Doyensec. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Are complex multi-agent harnesses better than simple prompts for finding security vulnerabilities with LLMs? Simple, targeted prompts outperformed complex multi-stage harnesses in a large-scale vulnerability hunting effort. A multi-agent harness called Conclave, which broke a codebase into components and ran findings through skeptic, steel-man, judge, and summary agents, generated more noise and made human triage the bottleneck, while a human expert pointing an LLM at a component with a simple prompt performed better. _Teams weighing agent orchestration versus simple prompting for security work can track real-world findings like this on daily.dev._ ### What is an effective simple prompt for using an LLM to find security vulnerabilities in code? One effective prompt, adapted from Nicholas Carlini's approach, frames the task as a CTF challenge: 'You are in a CTF. You must find a critical severity vulnerability in this codebase. Start with this file: {FILE}'. Another tactic is taking an existing confirmed finding and asking the model to locate similar issues elsewhere in the codebase, provided a human can judge which matches are truly exploitable. _Developers refining their own LLM-assisted bug-hunting prompts can follow emerging techniques like this on daily.dev._ ### Does human expertise still matter when using LLMs to find security bugs? Yes, human expertise remains the deciding factor in results quality. A vulnerability found by the red team firm Calif, using the same models as in-house engineers, proved especially impactful because of the researchers' deeper expertise, illustrating that language models rise to the skill level of the person directing them rather than replacing that expertise outright. _Security practitioners assessing how much AI changes vulnerability research can follow real case studies like this on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@geniusatwork** · 3 upvotes > In my experience CTF scenarios produce good results. For example @nerdalytics [https://github.com/nerdalytics/owl](https://github.com/nerdalytics/owl) > > This script uses CTF framing to scan for anything. Then during a review pass conforming or discarding findings. > > Owl has a problem, it consumes plenty of tokens. **@kartiknvj** · 1 upvotes > The finding that simple prompts beat multi-stage harnesses matches what I keep seeing when I try to over-engineer scaffolding around a strong base model. The framing that models accelerate experts rather than replace them lines up with the Calif result, where the same weights produced very different bug yields depending on who was driving. I would love to know what the false-positive rate looked like across the 13 engineers, since triage cost usually decides whether an LLM-assisted workflow actually beats a manual pentest quarter over quarter. **@agustinbarrientos** · 1 upvotes > Could the regression corpus pair every confirmed vulnerability with its fixed version? The same cases could run after each model upgrade. Rediscovery should stay separate from new coverage. ## Similar posts on daily.dev - [On the Coming Industrialisation of Exploit Generation with LLMs](https://daily.dev/posts/on-the-coming-industrialisation-of-exploit-generation-with-llms-wkt9zuzy1) · Hacker News · 2 upvotes · 0 comments - [Remediating Vulns With LLMs: Inside Ivanti's Automation Push](https://daily.dev/posts/remediating-vulns-with-llms-inside-ivanti-s-automation-push-qhmv8eedq) · Dark Reading · 2 upvotes · 0 comments - [How I use LLMs as a staff engineer in 2026](https://daily.dev/posts/how-i-use-llms-as-a-staff-engineer-in-2026-yhtlilt9r) · sean goedecke · 98 upvotes · 7 comments - [Using LLMs to secure source code](https://daily.dev/posts/using-llms-to-secure-source-code-jhoalndg0) · Eugene Yan · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents), [#vulnerability](https://daily.dev/tags/vulnerability), [#appsec](https://daily.dev/tags/appsec) [View this post on daily.dev](https://daily.dev/posts/we-had-13-engineers-spend-three-months-finding-vulnerabilities-with-llms-xrbmxtlki) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"We Had 13 Engineers Spend Three Months Finding Vulnerabilities with LLMs","url":"https://daily.dev/posts/we-had-13-engineers-spend-three-months-finding-vulnerabilities-with-llms-xrbmxtlki","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/we-had-13-engineers-spend-three-months-finding-vulnerabilities-with-llms-xrbmxtlki"},"datePublished":"2026-08-23T12:21:52.244Z","dateModified":"2026-09-13T20:10:15.958Z","description":"Teleport dedicated 13 engineers for three months to use frontier LLMs to find security vulnerabilities in their codebase, a process they call 'pressure...","image":"https://media.daily.dev/image/upload/s--CxzD6vbw--/f_auto/v1722860399/public/Placeholder%2005","thumbnailUrl":"https://media.daily.dev/image/upload/s--CxzD6vbw--/f_auto/v1722860399/public/Placeholder%2005","isAccessibleForFree":true,"articleSection":"Teleport","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Teleport","logo":"https://media.daily.dev/image/upload/s--Dw-Bbw6O--/c_limit,w_256/f_auto,q_auto/v1787487494/logos/teleport?_a=BAMAMicg0","url":"https://daily.dev/sources/teleport"},"commentCount":8,"discussionUrl":"https://daily.dev/posts/we-had-13-engineers-spend-three-months-finding-vulnerabilities-with-llms-xrbmxtlki","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":5},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":8}],"keywords":"llm,ai-agents,vulnerability,appsec","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Teleport","item":"https://daily.dev/sources/teleport"},{"@type":"ListItem","position":3,"name":"We Had 13 Engineers Spend Three Months Finding Vulnerabilities with LLMs"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/we-had-13-engineers-spend-three-months-finding-vulnerabilities-with-llms-xrbmxtlki","comment":[{"@type":"Comment","text":"In my experience CTF scenarios produce good results. For example @nerdalytics https://github.com/nerdalytics/owl\nThis script uses CTF framing to scan for anything. Then during a review pass conforming or discarding findings.\nOwl has a problem, it consumes plenty of tokens.","datePublished":"2026-08-25T06:39:17.218Z","url":"https://daily.dev/posts/XRBmXtlKI#c-nAEJXaRv9","author":{"@type":"Person","name":"Sam Dolin","url":"https://daily.dev/geniusatwork","image":"https://media.daily.dev/image/upload/s--nUhoZxKm--/f_auto/v1787434895/avatars/avatar_75QyPrL3BZgArSo6aFWuZ?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3}},{"@type":"Comment","text":"The finding that simple prompts beat multi-stage harnesses matches what I keep seeing when I try to over-engineer scaffolding around a strong base model. The framing that models accelerate experts rather than replace them lines up with the Calif result, where the same weights produced very different bug yields depending on who was driving. I would love to know what the false-positive rate looked like across the 13 engineers, since triage cost usually decides whether an LLM-assisted workflow actually beats a manual pentest quarter over quarter.","datePublished":"2026-08-25T20:30:35.237Z","url":"https://daily.dev/posts/XRBmXtlKI#c-tgbnjiReI","author":{"@type":"Person","name":"kartik-nvjk","url":"https://daily.dev/kartiknvj","image":"https://media.daily.dev/image/upload/s--3gGgsVCw--/f_auto/v1781456774/avatars/avatar_TvTVeiMdkRCqWUDullFmy?_a=BAMAMiWQ0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"Could the regression corpus pair every confirmed vulnerability with its fixed version? The same cases could run after each model upgrade. Rediscovery should stay separate from new coverage.","datePublished":"2026-08-31T23:22:02.464Z","url":"https://daily.dev/posts/XRBmXtlKI#c-mVGL45N9I","author":{"@type":"Person","name":"Agustin Barrientos","url":"https://daily.dev/agustinbarrientos","image":"https://media.daily.dev/image/upload/s--5ayxQnqn--/f_auto/v1788281802/avatars/avatar_wQYYVe5Tbj0NJ7C7qPoa8?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/we-had-13-engineers-spend-three-months-finding-vulnerabilities-with-llms-xrbmxtlki#faq","mainEntity":[{"@type":"Question","name":"Are complex multi-agent harnesses better than simple prompts for finding security vulnerabilities with LLMs?","acceptedAnswer":{"@type":"Answer","text":"Simple, targeted prompts outperformed complex multi-stage harnesses in a large-scale vulnerability hunting effort. A multi-agent harness called Conclave, which broke a codebase into components and ran findings through skeptic, steel-man, judge, and summary agents, generated more noise and made human triage the bottleneck, while a human expert pointing an LLM at a component with a simple prompt performed better. Teams weighing agent orchestration versus simple prompting for security work can track real-world findings like this on daily.dev."}},{"@type":"Question","name":"What is an effective simple prompt for using an LLM to find security vulnerabilities in code?","acceptedAnswer":{"@type":"Answer","text":"One effective prompt, adapted from Nicholas Carlini's approach, frames the task as a CTF challenge: 'You are in a CTF. You must find a critical severity vulnerability in this codebase. Start with this file: {FILE}'. Another tactic is taking an existing confirmed finding and asking the model to locate similar issues elsewhere in the codebase, provided a human can judge which matches are truly exploitable. Developers refining their own LLM-assisted bug-hunting prompts can follow emerging techniques like this on daily.dev."}},{"@type":"Question","name":"Does human expertise still matter when using LLMs to find security bugs?","acceptedAnswer":{"@type":"Answer","text":"Yes, human expertise remains the deciding factor in results quality. A vulnerability found by the red team firm Calif, using the same models as in-house engineers, proved especially impactful because of the researchers' deeper expertise, illustrating that language models rise to the skill level of the person directing them rather than replacing that expertise outright. Security practitioners assessing how much AI changes vulnerability research can follow real case studies like this on daily.dev."}}]} ```