--- title: "Two ways to measure the cumulative impact of experiments" url: https://daily.dev/posts/two-ways-to-measure-the-cumulative-impact-of-experiments-3tqdh1srd source_url: https://www.datadoghq.com/blog/two-ways-to-measure-cumulative-impact type: article source: "Datadog" published: 2026-08-18T14:48:05.874Z updated: 2026-08-18T15:05:53.000Z tags: ["devops", "statistics", "ab-testing"] reading_time: 9 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Two ways to measure the cumulative impact of experiments **[Datadog](https://daily.dev/sources/datadog)** · 9 min read · 0 upvotes · 0 comments ## Summary Summing the observed lift from winning A/B tests overstates real impact due to a statistical bias called the winner's curse, where noisy estimates that clear a significance bar are inflated on average. Two better approaches are described: a holdout, which randomizes users into control and winners groups to directly measure combined effect, and Datadog's Cumulative Impact feature, which applies empirical Bayes shrinkage to correct and aggregate existing experiment estimates without needing a holdout. Holdouts give a direct randomized estimate and can capture long-term effects and interactions but require advance planning, extended runtime, and traffic tradeoffs. Cumulative Impact is faster and works retroactively but relies on assumptions of exchangeable experiments, a stable effect distribution, and no interactions among treatments. Guidance is given for choosing between the two, and teams can combine both. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does summing the observed lift from multiple winning A/B tests overstate the true cumulative impact? Summing observed lifts overstates impact because of the winner's curse: significance testing filters on the noisy estimate rather than the true effect, so only experiments where sampling noise pushed the estimate above the significance bar survive. Since surviving winners are not a random sample, their estimates are inflated on average, and adding them together carries that inflation into the total. _daily.dev surfaces practical explainers like this for teams building rigorous experimentation programs._ ### What is a holdout group in A/B testing and how large should it be relative to the test population? A holdout group is a randomized segment of users kept on the pre-change product experience, used to measure the combined effect of all shipped winning variants against a separate winners-only group. Typical holdout and winners-only groups are each 1-5% of traffic, while the remainder, usually 90-98%, powers the ongoing A/B tests that surface winners. _Engineers designing experimentation infrastructure can track methodology guides like this on daily.dev._ ### What assumptions does Datadog's Cumulative Impact model rely on that a holdout does not need? Cumulative Impact relies on three assumptions: exchangeable experiments (all draw from the same effect distribution), a stable effect distribution across the analysis window, and no interactions among treatments so effects can be added or multiplied independently. It uses empirical Bayes shrinkage to correct estimates for the winner's curse without requiring a holdout, but pooling dissimilar experiments or shifting strategy mid-window breaks the correction. _daily.dev helps teams evaluating experimentation platforms compare tradeoffs like these before adopting a tool._ ## Similar posts on daily.dev - [Why Most A/B Tests Are Lying to You](https://daily.dev/posts/why-most-a-b-tests-are-lying-to-you-qelwpywxf) · Towards Data Science · 0 upvotes · 0 comments - [The effect distribution: The missing piece in experimentation programs](https://daily.dev/posts/the-effect-distribution-the-missing-piece-in-experimentation-programs-oxdrblabh) · Datadog · 0 upvotes · 0 comments --- Tags: [#devops](https://daily.dev/tags/devops), [#statistics](https://daily.dev/tags/statistics), [#ab-testing](https://daily.dev/tags/ab-testing) [View this post on daily.dev](https://daily.dev/posts/two-ways-to-measure-the-cumulative-impact-of-experiments-3tqdh1srd)