--- title: Four Ways a Test Can Lie to You | daily.dev description: Writing about 530 tests for an AI-assisted wine cellar app, the author found four distinct ways tests can pass or fail while proving nothing. One test asserted... canonical: https://daily.dev/posts/four-ways-a-test-can-lie-to-you-x5ipwk5wk twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Four Ways a Test Can Lie to You | daily.dev og:description: Writing about 530 tests for an AI-assisted wine cellar app, the author found four distinct ways tests can pass or fail while proving nothing. One test asserted... og:url: https://daily.dev/posts/four-ways-a-test-can-lie-to-you-x5ipwk5wk og:image: https://api.daily.dev/og/posts/x5IPWk5wK.png og:image:alt: Four Ways a Test Can Lie to You og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Four Ways a Test Can Lie to You **[DEV](https://daily.dev/sources/devto)** · 15 min read · 1 upvotes · 0 comments ## Summary Writing about 530 tests for an AI-assisted wine cellar app, the author found four distinct ways tests can pass or fail while proving nothing. One test asserted a rule had no effect but never actually exercised it, since a fixture's recency ordering masked the missing rule; deleting the rule left it green. A credential-leak grep flagged a legitimate export name on every clean build, so a check that always fails gets dismissed. A date-parsing test had correct positive and negative controls but encoded the wrong contract, since clamping is right only after the user finishes typing. A staging environment variable was silently ignored, so perfect assertions verified the wrong dataset. Fixes included paired positive/negative controls, mutation testing (delete the clause, watch it fail), and property tests across all 17,167 possible slider positions. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can a unit test pass even though it doesn't actually test the rule it claims to cover? A test can pass vacuously when its fixture already produces the expected outcome regardless of the rule under test. In one case, a test asserted that a proposed claim changes no bottle states, but the competing claim always won on recency whether proposed or accepted, so deleting the entire review-state filter from the source left the test green. The fix is a paired positive-control assertion proving the effect exists when the rule is enabled. _Anyone hardening a test suite against false confidence can track these verification patterns on daily.dev._ ### Why would a secret-scanning grep in CI keep failing on a build that has no leaked credentials? A regex pattern that's too loose can match legitimate strings. A credential grep using the pattern sk[A-Za-z0-9]{20,} fired on every clean build because it matched skipCrossDatasetReferenceValidation, a real export name in the @sanity/client package. Widening the minimum length from 20 to 40 characters fixed the false positive, since genuine credentials easily clear 40 characters but that export name doesn't. _Teams tuning CI security gates can compare notes on avoiding alert fatigue via daily.dev._ ### Why does typing a year into an HTML native date input field keep resetting to the minimum allowed date? A native date input only reports complete, well-formed dates, zero-padded to four digits, never partial text. Typing '2023' into a field constrained between 1996 and 2042 emits intermediate values like 0002-12-31 and 0202-12-31 before the final digit, and each one gets clamped to the 1996 floor, overwriting the field mid-edit. The underlying issue is that clamping out-of-range values is correct once editing is finished but wrong while the user is still typing. _Developers debugging date-input quirks can keep tabs on browser input behavior through daily.dev._ ## Similar posts on daily.dev - [Why AI Unit Tests Pass But Code Is Broken: The Test Agrees](https://daily.dev/posts/why-ai-unit-tests-pass-but-code-is-broken-the-test-agrees-diz6fcqc9) · Gil Zilberfeld - Everyday Unit Testing · 1 upvotes · 1 comments - [Why Cloud-Native Software Testing Requires Rethinking What a Passing Test Actually Means](https://daily.dev/posts/why-cloud-native-software-testing-requires-rethinking-what-a-passing-test-actually-means-yui7y8pjp) · Cloud Native Now · 0 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#webdev](https://daily.dev/tags/webdev), [#testing](https://daily.dev/tags/testing), [#ai-coding](https://daily.dev/tags/ai-coding), [#mutation-testing](https://daily.dev/tags/mutation-testing) [View this post on daily.dev](https://daily.dev/posts/four-ways-a-test-can-lie-to-you-x5ipwk5wk) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Four Ways a Test Can Lie to You","url":"https://daily.dev/posts/four-ways-a-test-can-lie-to-you-x5ipwk5wk","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/four-ways-a-test-can-lie-to-you-x5ipwk5wk"},"datePublished":"2026-10-30T17:36:11.170Z","dateModified":"2026-10-30T17:40:40.305Z","description":"Writing about 530 tests for an AI-assisted wine cellar app, the author found four distinct ways tests can pass or fail while proving nothing. One test asserted...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/7134bfe88cae607fae4202d97f9d82e9?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/7134bfe88cae607fae4202d97f9d82e9?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"DEV","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"DEV","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/devto","url":"https://daily.dev/sources/devto"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/four-ways-a-test-can-lie-to-you-x5ipwk5wk","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,webdev,testing,ai-coding,mutation-testing","timeRequired":"PT15M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"DEV","item":"https://daily.dev/sources/devto"},{"@type":"ListItem","position":3,"name":"Four Ways a Test Can Lie to You"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/four-ways-a-test-can-lie-to-you-x5ipwk5wk#faq","mainEntity":[{"@type":"Question","name":"How can a unit test pass even though it doesn't actually test the rule it claims to cover?","acceptedAnswer":{"@type":"Answer","text":"A test can pass vacuously when its fixture already produces the expected outcome regardless of the rule under test. In one case, a test asserted that a proposed claim changes no bottle states, but the competing claim always won on recency whether proposed or accepted, so deleting the entire review-state filter from the source left the test green. The fix is a paired positive-control assertion proving the effect exists when the rule is enabled. Anyone hardening a test suite against false confidence can track these verification patterns on daily.dev."}},{"@type":"Question","name":"Why would a secret-scanning grep in CI keep failing on a build that has no leaked credentials?","acceptedAnswer":{"@type":"Answer","text":"A regex pattern that's too loose can match legitimate strings. A credential grep using the pattern sk[A-Za-z0-9]{20,} fired on every clean build because it matched skipCrossDatasetReferenceValidation, a real export name in the @sanity/client package. Widening the minimum length from 20 to 40 characters fixed the false positive, since genuine credentials easily clear 40 characters but that export name doesn't. Teams tuning CI security gates can compare notes on avoiding alert fatigue via daily.dev."}},{"@type":"Question","name":"Why does typing a year into an HTML native date input field keep resetting to the minimum allowed date?","acceptedAnswer":{"@type":"Answer","text":"A native date input only reports complete, well-formed dates, zero-padded to four digits, never partial text. Typing '2023' into a field constrained between 1996 and 2042 emits intermediate values like 0002-12-31 and 0202-12-31 before the final digit, and each one gets clamped to the 1996 floor, overwriting the field mid-edit. The underlying issue is that clamping out-of-range values is correct once editing is finished but wrong while the user is still typing. Developers debugging date-input quirks can keep tabs on browser input behavior through daily.dev."}}]} ```