--- title: Catching Bugs Before They Land: JiTTesting’s... description: Meta has deployed Catching JiTTests, an LLM-based system that generates executable tests to catch bugs in code changes before they reach production. The system... canonical: https://daily.dev/posts/catching-bugs-before-they-land-jittesting-s-human-on-the-loop-approach-at-full-scale-bvjm0sq9j twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale | daily.dev og:description: Meta has deployed Catching JiTTests, an LLM-based system that generates executable tests to catch bugs in code changes before they reach production. The system... og:url: https://daily.dev/posts/catching-bugs-before-they-land-jittesting-s-human-on-the-loop-approach-at-full-scale-bvjm0sq9j og:image: https://api.daily.dev/og/posts/BVjM0SQ9j.png og:image:alt: Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale **[Facebook Engineering ](https://daily.dev/sources/facebook_code)** · 7 min read · 0 upvotes · 0 comments ## Summary Meta has deployed Catching JiTTests, an LLM-based system that generates executable tests to catch bugs in code changes before they reach production. The system uses two workflows: a 'dodgy diff' approach that treats every change as a potential bug, and an 'intent-aware' approach that infers a diff's purpose and likely risks. Weak catches are filtered through an LLM-as-Judge ensemble and a rule-based system called RubFake, then escalated to engineers using a human-on-the-loop model. 589 engineers (64% of outreach) have engaged with the system, reporting a 60% true-positive rate. In the first half of 2026 it caught 74 serious privacy bugs at the diff stage, cutting privacy SEV occurrences by 20%. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is Meta's JiTTests approach to catching bugs before deployment? JiTTests, or Just-in-Time Tests, is an LLM-based testing approach Meta uses to generate executable tests that expose behavioral differences introduced by a code change, rather than reviewing code like typical AI code review. It combines a 'dodgy diff' workflow that treats every change as a potential bug with an 'intent-aware' workflow that infers risks from the diff's intention, title, and summary. _Teams weighing AI-assisted testing strategies can track real deployment results like this one on daily.dev._ ### How effective was Meta's AI-generated test system (Catching JiTTests) in production? 589 engineers at Meta, or 64% of those reached out to, engaged with Catching JiTTests, and these engineers reported a 60% true-positive rate for the findings. In the first half of 2026 the system caught 74 serious privacy bugs at the diff stage and lowered privacy SEV occurrences by 20% during that period. _Developers evaluating AI testing tools against real adoption numbers can follow cases like this on daily.dev._ ### What is a human-on-the-loop approach versus human-in-the-loop for reviewing AI-generated test results? Human-on-the-loop means engineers set review criteria and check outcomes in aggregate rather than reviewing every single test result, unlike human-in-the-loop where a human reviews each output individually. Meta pairs this with an LLM-as-Judge ensemble and a rule-based filter called RubFake to rank weak catches before deciding whether to involve a human engineer at all. _Engineers designing review workflows for AI-generated code or tests can compare approaches like this on daily.dev._ ## Similar posts on daily.dev - [Meta Reports 4x Higher Bug Detection with Just-in-Time Testing](https://daily.dev/posts/meta-reports-4x-higher-bug-detection-with-just-in-time-testing-w7jc2qjj3) · InfoQ · 1 upvotes · 0 comments - [A Seasoned Tester's Crystal Ball: Capture the Bugs](https://daily.dev/posts/a-seasoned-tester-s-crystal-ball-capture-the-bugs-zppxnjhud) · A Seasoned Tester's Crystal Ball · 0 upvotes · 0 comments - [Don't Let the AI Find Your Bugs. Let It Judge Them.](https://daily.dev/posts/don-t-let-the-ai-find-your-bugs-let-it-judge-them--fpqkjl4lo) · Awesome Java Newsletter · 2 upvotes · 1 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#testing](https://daily.dev/tags/testing), [#ai-coding](https://daily.dev/tags/ai-coding), [#mutation-testing](https://daily.dev/tags/mutation-testing) [View this post on daily.dev](https://daily.dev/posts/catching-bugs-before-they-land-jittesting-s-human-on-the-loop-approach-at-full-scale-bvjm0sq9j) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale","url":"https://daily.dev/posts/catching-bugs-before-they-land-jittesting-s-human-on-the-loop-approach-at-full-scale-bvjm0sq9j","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/catching-bugs-before-they-land-jittesting-s-human-on-the-loop-approach-at-full-scale-bvjm0sq9j"},"datePublished":"2026-10-29T16:01:36.957Z","dateModified":"2026-10-29T16:04:51.309Z","description":"Meta has deployed Catching JiTTests, an LLM-based system that generates executable tests to catch bugs in code changes before they reach production. The system...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/b5425d2695406cec617cade04c746bfd?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/b5425d2695406cec617cade04c746bfd?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Facebook Engineering\n","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Facebook Engineering\n","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/facebook","url":"https://daily.dev/sources/facebook_code"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/catching-bugs-before-they-land-jittesting-s-human-on-the-loop-approach-at-full-scale-bvjm0sq9j","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,testing,ai-coding,mutation-testing","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Facebook Engineering\n","item":"https://daily.dev/sources/facebook_code"},{"@type":"ListItem","position":3,"name":"Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/catching-bugs-before-they-land-jittesting-s-human-on-the-loop-approach-at-full-scale-bvjm0sq9j#faq","mainEntity":[{"@type":"Question","name":"What is Meta's JiTTests approach to catching bugs before deployment?","acceptedAnswer":{"@type":"Answer","text":"JiTTests, or Just-in-Time Tests, is an LLM-based testing approach Meta uses to generate executable tests that expose behavioral differences introduced by a code change, rather than reviewing code like typical AI code review. It combines a 'dodgy diff' workflow that treats every change as a potential bug with an 'intent-aware' workflow that infers risks from the diff's intention, title, and summary. Teams weighing AI-assisted testing strategies can track real deployment results like this one on daily.dev."}},{"@type":"Question","name":"How effective was Meta's AI-generated test system (Catching JiTTests) in production?","acceptedAnswer":{"@type":"Answer","text":"589 engineers at Meta, or 64% of those reached out to, engaged with Catching JiTTests, and these engineers reported a 60% true-positive rate for the findings. In the first half of 2026 the system caught 74 serious privacy bugs at the diff stage and lowered privacy SEV occurrences by 20% during that period. Developers evaluating AI testing tools against real adoption numbers can follow cases like this on daily.dev."}},{"@type":"Question","name":"What is a human-on-the-loop approach versus human-in-the-loop for reviewing AI-generated test results?","acceptedAnswer":{"@type":"Answer","text":"Human-on-the-loop means engineers set review criteria and check outcomes in aggregate rather than reviewing every single test result, unlike human-in-the-loop where a human reviews each output individually. Meta pairs this with an LLM-as-Judge ensemble and a rule-based filter called RubFake to rank weak catches before deciding whether to involve a human engineer at all. Engineers designing review workflows for AI-generated code or tests can compare approaches like this on daily.dev."}}]} ```