--- title: The hardest part of catching AI cheats wasn't catching... description: Building an AI cheat detector for pull requests revealed that recall (catching bad code) is only half the problem — precision (avoiding false accusations) is... canonical: https://daily.dev/posts/the-hardest-part-of-catching-ai-cheats-wasn-t-catching-them-it-was-not-crying-wolf--ofqqwjclj twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: The hardest part of catching AI cheats wasn't catching them. It was not crying wolf. | daily.dev og:description: Building an AI cheat detector for pull requests revealed that recall (catching bad code) is only half the problem — precision (avoiding false accusations) is... og:url: https://daily.dev/posts/the-hardest-part-of-catching-ai-cheats-wasn-t-catching-them-it-was-not-crying-wolf--ofqqwjclj og:image: https://api.daily.dev/og/posts/OfQQWJcLj.png og:image:alt: The hardest part of catching AI cheats wasn't catching them. It was not crying wolf. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # The hardest part of catching AI cheats wasn't catching them. It was not crying wolf. **[Deleted user](https://daily.dev/sources/404)** · [@ghost](https://daily.dev/ghost) · 2 min read · 0 upvotes · 0 comments ## Summary Building an AI cheat detector for pull requests revealed that recall (catching bad code) is only half the problem — precision (avoiding false accusations) is what determines whether anyone uses the tool at all. An early pilot produced 3.17 false alarms per PR, which is enough to make muting the correct rational response. The author shifted focus to refuters, confirmation gates, and benching detectors that never fired on real PRs. The result was 0.11 false alarms per PR across 18 PRs on five public repos. The tool stays advisory by default because a false block costs a merge and creates adversarial relationships; blocking mode requires measured 0.90+ precision with statistical confidence before activating. The core insight: the real budget isn't compute, it's the reader's attention, and every false positive spends it. ## Content Four detectors in my PR auditor produced 481 findings on a scan of real merged PRs. Confirmed cheats after triage: zero. Everything I'd published about the project before that was about recall. How much it catches. Triaging those 481 findings showed me the other cost: nobody argues with a noisy tool. They mute it. After that, recall is irrelevant. The early pilot measured 3.17 false alarms per PR. At that rate, muting isn't laziness. It's the correct response. Worse, recall work makes precision worse. Every new detector is also a new source of false accusations. One of mine shipped with its judge logic inverted: it raised the alarm exactly when the judge said the fix was fine. I only caught it by triaging false alarms. Every benchmark before that asked what the tool caught, never what it wrongly accused. So the work shifted. Refuters that drop a finding when the diff itself proves the pattern legitimate. A confirmation gate before any finding carries weight. Three detectors benched entirely because they never fired on a real PR. Result: 0.11 false alarms per PR across an 18-PR pilot on five public repos (vite, next.js, astro, nx, trpc). I used to quote that as a stat. It's really a survival threshold. Below it, people read every finding. Above it, the tool is wallpaper. It's also why the tool is advisory by default. A false flag costs a minute. A false block costs a merge and makes an enemy. Blocking stays off until measured precision clears a 0.90 floor with statistical confidence, and the data doesn't prove that yet, so the gate says so itself. Your real budget isn't compute. It's the reader's attention, and every false positive spends it. Repo: https://github.com/moonrunnerkc/swarm-orchestrator ## Similar posts on daily.dev - [Why AI Code Reviews Are Slowing Down PRs](https://daily.dev/posts/why-ai-code-reviews-are-slowing-down-prs-yubjqxjqr) · Aviator Blog · 0 upvotes · 1 comments - [Stop Sending IDE-Catchable AI Code Errors to Review](https://daily.dev/posts/stop-sending-ide-catchable-ai-code-errors-to-review-lepmps4fa) · JetBrains · 0 upvotes · 0 comments - [The AI Pair Programmer Your Team Doesn't Trust, CodeGood](https://daily.dev/posts/the-ai-pair-programmer-your-team-doesn-t-trust-codegood-cajoas9xd) · CodeGood · 1 upvotes · 1 comments --- Tags: [#devtools](https://daily.dev/tags/devtools), [#ai-coding](https://daily.dev/tags/ai-coding) [View this post on daily.dev](https://daily.dev/posts/the-hardest-part-of-catching-ai-cheats-wasn-t-catching-them-it-was-not-crying-wolf--ofqqwjclj) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/the-hardest-part-of-catching-ai-cheats-wasn-t-catching-them-it-was-not-crying-wolf--ofqqwjclj","headline":"The hardest part of catching AI cheats wasn't catching them. It was not crying wolf.","text":"Building an AI cheat detector for pull requests revealed that recall (catching bad code) is only half the problem — precision (avoiding false accusations) is what determines whether anyone uses the tool at all. An early pilot produced 3.17 false alarms per PR, which is enough to make muting the correct rational response. The author shifted focus to refuters, confirmation gates, and benching detectors that never fired on real PRs. The result was 0.11 false alarms per PR across 18 PRs on five public repos. The tool stays advisory by default because a false block costs a merge and creates adversarial relationships; blocking mode requires measured 0.90+ precision with statistical confidence before activating. The core insight: the real budget isn't compute, it's the reader's attention, and every false positive spends it.","url":"https://daily.dev/posts/the-hardest-part-of-catching-ai-cheats-wasn-t-catching-them-it-was-not-crying-wolf--ofqqwjclj","datePublished":"2026-07-23T21:03:21.780Z","dateModified":"2026-07-23T21:03:39.583Z","author":{"@type":"Person","name":"Deleted user","url":"https://daily.dev/ghost","image":"https://media.daily.dev/image/upload/s--hNIUzLiO--/f_auto/v1705327420/public/ghost_vlftth","interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":34950}},"interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/404","name":"Deleted user"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Deleted user","item":"https://daily.dev/sources/404"},{"@type":"ListItem","position":3,"name":"The hardest part of catching AI cheats wasn't catching them. It was not crying wolf."}]} ```