--- title: The Test Passed But The Product Broke Anyway. WhatIF…? description: WhatIF V2 is an open-source adversarial regression and requirements engine that addresses a fundamental gap in software testing: acceptance criteria only... canonical: https://daily.dev/posts/the-test-passed-but-the-product-broke-anyway-whatif--myuvnlrlk twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: The Test Passed But The Product Broke Anyway. WhatIF…? | daily.dev og:description: WhatIF V2 is an open-source adversarial regression and requirements engine that addresses a fundamental gap in software testing: acceptance criteria only... og:url: https://daily.dev/posts/the-test-passed-but-the-product-broke-anyway-whatif--myuvnlrlk og:image: https://api.daily.dev/og/posts/mYUVNLRlk.png og:image:alt: The Test Passed But The Product Broke Anyway. WhatIF…? og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # The Test Passed But The Product Broke Anyway. WhatIF…? **[Medium](https://daily.dev/sources/medium_js)** · 11 min read · 0 upvotes · 0 comments ## Summary WhatIF V2 is an open-source adversarial regression and requirements engine that addresses a fundamental gap in software testing: acceptance criteria only verify what was intended, not what was accidentally broken. The tool uses static analysis to map dependency relationships in a codebase first, then applies AI reasoning over that evidence to identify the 'blast radius' of a change — the downstream effects on authentication, billing, CRM sync, and other systems that were never mentioned in the ticket. Unlike generic AI test generators, WhatIF V2 enforces an 'evidence before imagination' rule, preventing hallucinated dependencies. It classifies findings by testability (auto-executable, semi-automatic, manual, architectural, or insufficient evidence) and generates standard pytest probes rather than a proprietary DSL. Analysis and execution are deliberately separated via `whatif analyze` and `whatif test` commands. The AI provider (Ollama, OpenAI, Claude) is replaceable and hosted providers are explicitly opt-in to protect source code privacy. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is the difference between WhatIF and QScript in testing a codebase? QScript turns acceptance criteria into an executable contract that verifies a feature does what was intended, while WhatIF looks outward from a code change to find regressions and forgotten edge cases the acceptance criteria never specified. WhatIF can read .qscript contracts as evidence of what is already covered, letting it focus more on blast-radius analysis rather than re-testing the happy path. Neither tool depends on the other. _Comparing regression tools like this is easier when developers track new testing approaches on daily.dev._ ### How does WhatIF V2 prevent an AI model from hallucinating fake dependencies when analyzing a codebase? WhatIF V2 enforces an 'evidence before imagination' architecture: static analysis first maps real dependency relationships from the repository, and only after that evidence is established does the AI reason about possible consequences. This replaces the original version, which let an LLM freely inspect a repo and invent counterfactual failure scenarios without grounding them in actual code relationships. _Developers evaluating AI-assisted testing tools can follow architecture changes like this via daily.dev._ ### Does WhatIF execute the tests it generates automatically? No, analysis and execution are deliberately separate commands. Running 'whatif analyze .' only produces findings and does not execute any generated tests; a separate 'whatif test' command is required to run any safe executable probes, which are written as ordinary pytest tests using httpx for APIs or Playwright for browser regressions. _Anyone weighing test-automation workflows can keep tabs on tools like this through daily.dev._ ## Similar posts on daily.dev - [The AI-augmented tester is here. So is a new problem: proving your tests actually work](https://daily.dev/posts/the-ai-augmented-tester-is-here-so-is-a-new-problem-proving-your-tests-actually-work-wnh2mhwcr) · SD Times · 0 upvotes · 1 comments - [Toward automated verification of unreviewed AI-generated code](https://daily.dev/posts/toward-automated-verification-of-unreviewed-ai-generated-code-iogcc4ffa) · Hacker News · 0 upvotes · 0 comments - [Why AI Unit Tests Pass But Code Is Broken: The Test Agrees](https://daily.dev/posts/why-ai-unit-tests-pass-but-code-is-broken-the-test-agrees-diz6fcqc9) · Gil Zilberfeld - Everyday Unit Testing · 1 upvotes · 1 comments - [I Almost Built an AI QA Engineer. Then I Did the Math and Built the Boring Version Instead.](https://daily.dev/posts/i-almost-built-an-ai-qa-engineer-then-i-did-the-math-and-built-the-boring-version-instead--6bkbbxthi) · Medium · 0 upvotes · 0 comments --- Tags: [#python](https://daily.dev/tags/python), [#testing](https://daily.dev/tags/testing) [View this post on daily.dev](https://daily.dev/posts/the-test-passed-but-the-product-broke-anyway-whatif--myuvnlrlk) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"The Test Passed But The Product Broke Anyway. WhatIF…?","url":"https://daily.dev/posts/the-test-passed-but-the-product-broke-anyway-whatif--myuvnlrlk","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/the-test-passed-but-the-product-broke-anyway-whatif--myuvnlrlk"},"datePublished":"2026-08-11T13:48:33.874Z","dateModified":"2026-09-14T07:53:37.723Z","description":"WhatIF V2 is an open-source adversarial regression and requirements engine that addresses a fundamental gap in software testing: acceptance criteria only...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/65633214dc0dc93e2ed883251a96a8be?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/65633214dc0dc93e2ed883251a96a8be?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/the-test-passed-but-the-product-broke-anyway-whatif--myuvnlrlk","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,testing","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"The Test Passed But The Product Broke Anyway. WhatIF…?"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/the-test-passed-but-the-product-broke-anyway-whatif--myuvnlrlk#faq","mainEntity":[{"@type":"Question","name":"What is the difference between WhatIF and QScript in testing a codebase?","acceptedAnswer":{"@type":"Answer","text":"QScript turns acceptance criteria into an executable contract that verifies a feature does what was intended, while WhatIF looks outward from a code change to find regressions and forgotten edge cases the acceptance criteria never specified. WhatIF can read .qscript contracts as evidence of what is already covered, letting it focus more on blast-radius analysis rather than re-testing the happy path. Neither tool depends on the other. Comparing regression tools like this is easier when developers track new testing approaches on daily.dev."}},{"@type":"Question","name":"How does WhatIF V2 prevent an AI model from hallucinating fake dependencies when analyzing a codebase?","acceptedAnswer":{"@type":"Answer","text":"WhatIF V2 enforces an 'evidence before imagination' architecture: static analysis first maps real dependency relationships from the repository, and only after that evidence is established does the AI reason about possible consequences. This replaces the original version, which let an LLM freely inspect a repo and invent counterfactual failure scenarios without grounding them in actual code relationships. Developers evaluating AI-assisted testing tools can follow architecture changes like this via daily.dev."}},{"@type":"Question","name":"Does WhatIF execute the tests it generates automatically?","acceptedAnswer":{"@type":"Answer","text":"No, analysis and execution are deliberately separate commands. Running 'whatif analyze .' only produces findings and does not execute any generated tests; a separate 'whatif test' command is required to run any safe executable probes, which are written as ordinary pytest tests using httpx for APIs or Playwright for browser regressions. Anyone weighing test-automation workflows can keep tabs on tools like this through daily.dev."}}]} ```