Summing the observed lift from winning A/B tests overstates real impact due to a statistical bias called the winner's curse, where noisy estimates that clear a significance bar are inflated on average. Two better approaches are described: a holdout, which randomizes users into control and winners groups to directly measure combined effect, and Datadog's Cumulative Impact feature, which applies empirical Bayes shrinkage to correct and aggregate existing experiment estimates without needing a holdout. Holdouts give a direct randomized estimate and can capture long-term effects and interactions but require advance planning, extended runtime, and traffic tradeoffs. Cumulative Impact is faster and works retroactively but relies on assumptions of exchangeable experiments, a stable effect distribution, and no interactions among treatments. Guidance is given for choosing between the two, and teams can combine both.
Table of contents
Why naive summation overstates cumulative impactMeasure cumulative impact directly with a holdoutEstimate cumulative impact without a holdout using Cumulative ImpactChoose between a holdout and Cumulative ImpactKey takeaways for measuring cumulative impactQuestions this post answers
Why does summing the observed lift from multiple winning A/B tests overstate the true cumulative impact?
Summing observed lifts overstates impact because of the winner's curse: significance testing filters on the noisy estimate rather than the true effect, so only experiments where sampling noise pushed the estimate above the significance bar survive. Since surviving winners are not a random sample, their estimates are inflated on average, and adding them together carries that inflation into the total. daily.dev surfaces practical explainers like this for teams building rigorous experimentation programs.
What is a holdout group in A/B testing and how large should it be relative to the test population?
A holdout group is a randomized segment of users kept on the pre-change product experience, used to measure the combined effect of all shipped winning variants against a separate winners-only group. Typical holdout and winners-only groups are each 1-5% of traffic, while the remainder, usually 90-98%, powers the ongoing A/B tests that surface winners. Engineers designing experimentation infrastructure can track methodology guides like this on daily.dev.
What assumptions does Datadog's Cumulative Impact model rely on that a holdout does not need?
Cumulative Impact relies on three assumptions: exchangeable experiments (all draw from the same effect distribution), a stable effect distribution across the analysis window, and no interactions among treatments so effects can be added or multiplied independently. It uses empirical Bayes shrinkage to correct estimates for the winner's curse without requiring a holdout, but pooling dissimilar experiments or shifting strategy mid-window breaks the correction. daily.dev helps teams evaluating experimentation platforms compare tradeoffs like these before adopting a tool.