--- title: Stop Calling the First Significant Day a Win | daily.dev description: Daily peeking at A/B test dashboards inflates the nominal 5% false-positive rate to 27.7% when checked every day for 30 days. A seeded Python simulation... canonical: https://daily.dev/posts/stop-calling-the-first-significant-day-a-win-u17yzbxeg twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Stop Calling the First Significant Day a Win | daily.dev og:description: Daily peeking at A/B test dashboards inflates the nominal 5% false-positive rate to 27.7% when checked every day for 30 days. A seeded Python simulation... og:url: https://daily.dev/posts/stop-calling-the-first-significant-day-a-win-u17yzbxeg og:image: https://api.daily.dev/og/posts/U17yzBXEG.png og:image:alt: Stop Calling the First Significant Day a Win og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Stop Calling the First Significant Day a Win **[Towards Data Science](https://daily.dev/sources/tds)** · 12 min read · 0 upvotes · 0 comments ## Summary Daily peeking at A/B test dashboards inflates the nominal 5% false-positive rate to 27.7% when checked every day for 30 days. A seeded Python simulation demonstrates this inflation across different peeking frequencies (2 looks → 8.3%, 5 looks → 14%, 10 looks → 19.1%, 30 looks → 27.7%). Even when a real effect exists, early stopping exaggerates the measured lift — a true 10% lift appears as 12.7% when stopped at first significance. Three corrective approaches are benchmarked: fixed-sample no-peeking (5.1% FPR, 97.9% power, 30 days), Pocock group-sequential boundary with z=2.73 (4.9% FPR, 93.3% power, ~11 days), and always-valid p-values via mSPRT (1.5% FPR, 87.5% power, ~14 days). The key recommendation is to declare the stopping rule before the test starts and choose a method that matches how the team actually monitors experiments. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much does daily peeking inflate the false-positive rate in an A/B test run for 30 days? Daily peeking at an A/B test for 30 days inflates the false-positive rate from the nominal 5% to 27.7%, meaning more than one in four 'wins' are artifacts of the stopping rule rather than real effects. The inflation is progressive: 2 looks yields 8.3%, 5 looks 14.0%, and 10 looks 19.1%, all simulated with two identical arms converting at 10%. _Teams shipping experiments without a declared stopping rule keep hitting this problem — developers tracking experimentation rigor discuss it on daily.dev._ ### What is the Pocock boundary z-cutoff for daily peeking over 30 days in an A/B test and how does it affect power? A Pocock-style boundary calibrated for daily peeking over 30 days uses a z-cutoff of 2.73 instead of the usual 1.96. This holds the false-positive rate at 4.9% while retaining 93.3% power against a true 10% relative lift, with a typical decision by day 11 — compared to day 30 for a fixed-sample design with 97.9% power. _Choosing between sequential methods and fixed-sample designs is exactly the kind of tradeoff experimentation engineers weigh on daily.dev._ ### Why does stopping an A/B test at first significance exaggerate the measured lift even when the effect is real? Stopping at first significance requires an unusually favorable random swing to cross the threshold early, so the estimate at that moment is biased upward. A simulation with a true 10% relative lift (A at 10%, B at 11% conversion) showed a fixed 30-day horizon recorded a median lift of 10.1%, while stopping at first significance recorded 12.7% — about a quarter higher than the true effect. _Inflated lift estimates feed into revenue forecasts and roadmap decisions, a risk product teams building on experiment results track on daily.dev._ ## Similar posts on daily.dev - [Product Experimentation: Stop Early Without P-Hacking Using mSPRT and Sequential Testing in Python](https://daily.dev/posts/product-experimentation-stop-early-without-p-hacking-using-msprt-and-sequential-testing-in-python-kqvlqrnu7) · freeCodeCamp · 1 upvotes · 0 comments - [Why Most A/B Tests Are Lying to You](https://daily.dev/posts/why-most-a-b-tests-are-lying-to-you-qelwpywxf) · Towards Data Science · 0 upvotes · 0 comments - [Designing a Point-in-Time Backtest You Can Actually Trust](https://daily.dev/posts/designing-a-point-in-time-backtest-you-can-actually-trust-ae4li3wyj) · Medium · 3 upvotes · 0 comments - [Two ways to measure the cumulative impact of experiments](https://daily.dev/posts/two-ways-to-measure-the-cumulative-impact-of-experiments-3tqdh1srd) · Datadog · 0 upvotes · 0 comments --- Tags: [#python](https://daily.dev/tags/python), [#data-science](https://daily.dev/tags/data-science), [#ab-testing](https://daily.dev/tags/ab-testing) [View this post on daily.dev](https://daily.dev/posts/stop-calling-the-first-significant-day-a-win-u17yzbxeg) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Stop Calling the First Significant Day a Win","url":"https://daily.dev/posts/stop-calling-the-first-significant-day-a-win-u17yzbxeg","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/stop-calling-the-first-significant-day-a-win-u17yzbxeg"},"datePublished":"2026-08-11T16:39:07.439Z","dateModified":"2026-08-11T16:39:51.457Z","description":"Daily peeking at A/B test dashboards inflates the nominal 5% false-positive rate to 27.7% when checked every day for 30 days. A seeded Python simulation...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/36dfe0ca6e17c632a591421cd2cde267?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/36dfe0ca6e17c632a591421cd2cde267?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Towards Data Science","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Towards Data Science","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/tds","url":"https://daily.dev/sources/tds"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/stop-calling-the-first-significant-day-a-win-u17yzbxeg","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,data-science,ab-testing","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Towards Data Science","item":"https://daily.dev/sources/tds"},{"@type":"ListItem","position":3,"name":"Stop Calling the First Significant Day a Win"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/stop-calling-the-first-significant-day-a-win-u17yzbxeg#faq","mainEntity":[{"@type":"Question","name":"How much does daily peeking inflate the false-positive rate in an A/B test run for 30 days?","acceptedAnswer":{"@type":"Answer","text":"Daily peeking at an A/B test for 30 days inflates the false-positive rate from the nominal 5% to 27.7%, meaning more than one in four 'wins' are artifacts of the stopping rule rather than real effects. The inflation is progressive: 2 looks yields 8.3%, 5 looks 14.0%, and 10 looks 19.1%, all simulated with two identical arms converting at 10%. Teams shipping experiments without a declared stopping rule keep hitting this problem — developers tracking experimentation rigor discuss it on daily.dev."}},{"@type":"Question","name":"What is the Pocock boundary z-cutoff for daily peeking over 30 days in an A/B test and how does it affect power?","acceptedAnswer":{"@type":"Answer","text":"A Pocock-style boundary calibrated for daily peeking over 30 days uses a z-cutoff of 2.73 instead of the usual 1.96. This holds the false-positive rate at 4.9% while retaining 93.3% power against a true 10% relative lift, with a typical decision by day 11 — compared to day 30 for a fixed-sample design with 97.9% power. Choosing between sequential methods and fixed-sample designs is exactly the kind of tradeoff experimentation engineers weigh on daily.dev."}},{"@type":"Question","name":"Why does stopping an A/B test at first significance exaggerate the measured lift even when the effect is real?","acceptedAnswer":{"@type":"Answer","text":"Stopping at first significance requires an unusually favorable random swing to cross the threshold early, so the estimate at that moment is biased upward. A simulation with a true 10% relative lift (A at 10%, B at 11% conversion) showed a fixed 30-day horizon recorded a median lift of 10.1%, while stopping at first significance recorded 12.7% — about a quarter higher than the true effect. Inflated lift estimates feed into revenue forecasts and roadmap decisions, a risk product teams building on experiment results track on daily.dev."}}]} ```