--- title: Two ways to measure the cumulative impact of experiments description: Summing the observed lift from winning A/B tests overstates real impact due to a statistical bias called the winner's curse, where noisy estimates that clear a... canonical: https://daily.dev/posts/two-ways-to-measure-the-cumulative-impact-of-experiments-3tqdh1srd twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Two ways to measure the cumulative impact of experiments | daily.dev og:description: Summing the observed lift from winning A/B tests overstates real impact due to a statistical bias called the winner's curse, where noisy estimates that clear a... og:url: https://daily.dev/posts/two-ways-to-measure-the-cumulative-impact-of-experiments-3tqdh1srd og:image: https://api.daily.dev/og/posts/3TqDH1SRD.png og:image:alt: Two ways to measure the cumulative impact of experiments og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Two ways to measure the cumulative impact of experiments **[Datadog](https://daily.dev/sources/datadog)** · 9 min read · 0 upvotes · 0 comments ## Summary Summing the observed lift from winning A/B tests overstates real impact due to a statistical bias called the winner's curse, where noisy estimates that clear a significance bar are inflated on average. Two better approaches are described: a holdout, which randomizes users into control and winners groups to directly measure combined effect, and Datadog's Cumulative Impact feature, which applies empirical Bayes shrinkage to correct and aggregate existing experiment estimates without needing a holdout. Holdouts give a direct randomized estimate and can capture long-term effects and interactions but require advance planning, extended runtime, and traffic tradeoffs. Cumulative Impact is faster and works retroactively but relies on assumptions of exchangeable experiments, a stable effect distribution, and no interactions among treatments. Guidance is given for choosing between the two, and teams can combine both. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### why does summing the lift from multiple winning A/B tests overstate the true cumulative impact Summing observed lifts overstates impact because of the winner's curse: significance testing filters on the noisy estimate rather than the true effect, so only experiments where sampling noise pushed the estimate above the significance bar survive. Because the surviving 'winners' are not a random sample, their estimates are inflated, and adding them together carries that inflation into the total. _Teams reporting experimentation ROI can dig into bias-correction methods like this one on daily.dev._ ### what is a holdout group in A/B testing and how is it structured A holdout group is a randomized subset of users, typically 1-5% of traffic, frozen on the pre-holdout product experience while a separate 'winners only' group of similar size receives every shipped winning variant. The remaining 90-98% of traffic runs the ongoing A/B tests that surface those winners. Comparing the holdout and winners-only groups gives a direct, unbiased estimate of the cumulative effect of everything shipped. _Anyone designing a holdout experiment can compare traffic-allocation tradeoffs like these on daily.dev._ ### what assumptions does Datadog's Cumulative Impact model rely on instead of running a holdout Cumulative Impact assumes exchangeable experiments (all draw from the same effect distribution), a stable effect distribution across the analysis window, and no interactions among treatments, correcting each experiment's estimate with empirical Bayes shrinkage rather than measuring a real control group. It needs at least five qualifying, comparable experiments to produce a stable estimate, trading a direct randomized measurement for a faster, model-based one. _Developers deciding between holdouts and model-based estimates can weigh these tradeoffs via daily.dev._ ## Similar posts on daily.dev - [Why Most A/B Tests Are Lying to You](https://daily.dev/posts/why-most-a-b-tests-are-lying-to-you-qelwpywxf) · Towards Data Science · 0 upvotes · 0 comments - [The effect distribution: The missing piece in experimentation programs](https://daily.dev/posts/the-effect-distribution-the-missing-piece-in-experimentation-programs-oxdrblabh) · Datadog · 0 upvotes · 0 comments - [Measuring Marketing Incrementality](https://daily.dev/posts/measuring-marketing-incrementality-c7sxndczj) · Medium · 0 upvotes · 0 comments - [Beyond A/B Testing: Using Surrogacy and Region-Splits to Measure Long-Term Effects in Marketplaces](https://daily.dev/posts/beyond-a-b-testing-using-surrogacy-and-region-splits-to-measure-long-term-effects-in-marketplaces-xjli28ikv) · Lyft Engineering · 0 upvotes · 0 comments - [Your AI Adoption Lift Is a Selection Effect](https://daily.dev/posts/your-ai-adoption-lift-is-a-selection-effect-llwfesyrj) · Towards Data Science · 2 upvotes · 0 comments --- Tags: [#devops](https://daily.dev/tags/devops), [#statistics](https://daily.dev/tags/statistics), [#ab-testing](https://daily.dev/tags/ab-testing) [View this post on daily.dev](https://daily.dev/posts/two-ways-to-measure-the-cumulative-impact-of-experiments-3tqdh1srd) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Two ways to measure the cumulative impact of experiments","url":"https://daily.dev/posts/two-ways-to-measure-the-cumulative-impact-of-experiments-3tqdh1srd","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/two-ways-to-measure-the-cumulative-impact-of-experiments-3tqdh1srd"},"datePublished":"2026-08-18T14:48:05.874Z","dateModified":"2026-09-14T07:43:09.709Z","description":"Summing the observed lift from winning A/B tests overstates real impact due to a statistical bias called the winner's curse, where noisy estimates that clear a...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2940ca6017bb37bba483c8329ef4c96e?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2940ca6017bb37bba483c8329ef4c96e?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Datadog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Datadog","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/8f9d7e3e4bd64bd7b9db9b2ea83e0c6a","url":"https://daily.dev/sources/datadog"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/two-ways-to-measure-the-cumulative-impact-of-experiments-3tqdh1srd","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"devops,statistics,ab-testing","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Datadog","item":"https://daily.dev/sources/datadog"},{"@type":"ListItem","position":3,"name":"Two ways to measure the cumulative impact of experiments"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/two-ways-to-measure-the-cumulative-impact-of-experiments-3tqdh1srd#faq","mainEntity":[{"@type":"Question","name":"why does summing the lift from multiple winning A/B tests overstate the true cumulative impact","acceptedAnswer":{"@type":"Answer","text":"Summing observed lifts overstates impact because of the winner's curse: significance testing filters on the noisy estimate rather than the true effect, so only experiments where sampling noise pushed the estimate above the significance bar survive. Because the surviving 'winners' are not a random sample, their estimates are inflated, and adding them together carries that inflation into the total. Teams reporting experimentation ROI can dig into bias-correction methods like this one on daily.dev."}},{"@type":"Question","name":"what is a holdout group in A/B testing and how is it structured","acceptedAnswer":{"@type":"Answer","text":"A holdout group is a randomized subset of users, typically 1-5% of traffic, frozen on the pre-holdout product experience while a separate 'winners only' group of similar size receives every shipped winning variant. The remaining 90-98% of traffic runs the ongoing A/B tests that surface those winners. Comparing the holdout and winners-only groups gives a direct, unbiased estimate of the cumulative effect of everything shipped. Anyone designing a holdout experiment can compare traffic-allocation tradeoffs like these on daily.dev."}},{"@type":"Question","name":"what assumptions does Datadog's Cumulative Impact model rely on instead of running a holdout","acceptedAnswer":{"@type":"Answer","text":"Cumulative Impact assumes exchangeable experiments (all draw from the same effect distribution), a stable effect distribution across the analysis window, and no interactions among treatments, correcting each experiment's estimate with empirical Bayes shrinkage rather than measuring a real control group. It needs at least five qualifying, comparable experiments to produce a stable estimate, trading a direct randomized measurement for a faster, model-based one. Developers deciding between holdouts and model-based estimates can weigh these tradeoffs via daily.dev."}}]} ```