--- title: Scaling Reliable Experimentation in a Two-Sided AdTech... description: Zalando's ad marketplace team describes how they solved statistical validity problems in A/B testing for their two-sided AdTech marketplace (ZMS). Because... canonical: https://daily.dev/posts/scaling-reliable-experimentation-in-a-two-sided-adtech-marketplace-zms-budget-split-segb1krmo twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Scaling Reliable Experimentation in a Two-Sided AdTech Marketplace: ZMS Budget Split | daily.dev og:description: Zalando's ad marketplace team describes how they solved statistical validity problems in A/B testing for their two-sided AdTech marketplace (ZMS). Because... og:url: https://daily.dev/posts/scaling-reliable-experimentation-in-a-two-sided-adtech-marketplace-zms-budget-split-segb1krmo og:image: https://api.daily.dev/og/posts/sEGB1Krmo.png og:image:alt: Scaling Reliable Experimentation in a Two-Sided AdTech Marketplace: ZMS Budget Split og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Scaling Reliable Experimentation in a Two-Sided AdTech Marketplace: ZMS Budget Split **[Zalando](https://daily.dev/sources/zalando)** · 5 min read · 0 upvotes · 0 comments ## Summary Zalando's ad marketplace team describes how they solved statistical validity problems in A/B testing for their two-sided AdTech marketplace (ZMS). Because control and treatment groups shared advertiser budgets, faster variants could cannibalize budget from slower ones, biasing results (a SUTVA violation called Cannibalization Bias). Their fix, Budget Split, allocates separate proportional budget pools per variant, creating isolated 'sub-campaigns' so variants no longer compete for the same money. To run multiple experiments simultaneously, they extended this with Orthogonal Concurrency, splitting budgets into four orthogonal buckets for two concurrent A/B experiments. They also outline residual sources of bias they still monitor, such as algorithmic budget steering, ML training data leakage, manual ops interventions, and censoring effects. The result: experimentation volume scaled from 8 experiments in 2023 to over 60 in 2025. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why do standard A/B tests give biased results in a two-sided advertising marketplace with shared budgets? Because control and treatment variants draw from the same finite advertiser budget, violating the Stable Unit Treatment Value Assumption (SUTVA). A more efficient variant wins more auctions and depletes the shared budget faster, starving the other variant of funding and artificially depressing its metrics, a phenomenon called cannibalization bias that dilutes the true average treatment effect. _daily.dev surfaces experimentation design write-ups for teams debugging biased marketplace test results._ ### What is Budget Split and how does it fix cannibalization bias in ad experiments? Budget Split is a technique that partitions each ad campaign's budget into separate virtual sub-campaigns, one per experiment variant, proportional to traffic allocation. Treatment users only trigger bids against the treatment sub-budget and control users against the control sub-budget, creating two isolated marketplaces so that outperformance reflects the feature itself rather than budget cannibalization. Zalando used this to scale from 8 experiments in 2023 to over 60 by 2025. _Track marketplace experimentation architecture patterns like this on daily.dev when designing your own test isolation._ ### How do you run multiple concurrent A/B experiments in the same ad marketplace without them interfering? Orthogonal Concurrency splits a campaign's budget into orthogonal buckets for each combination of variants across simultaneous experiments. For two concurrent experiments with A/B variants each, the budget is divided into four buckets (1A2A, 1A2B, 1B2A, 1B2B), preventing experiments from fighting over the same traffic and budget and enabling reliable concurrent testing. _daily.dev collects engineering deep dives like this for teams scaling concurrent experimentation platforms._ ## Similar posts on daily.dev - [A/B test models in production](https://daily.dev/posts/a-b-test-models-in-production-w0dbvgkzs) · Together AI · 0 upvotes · 0 comments - [Large scale A/B experimentation on data center infrastructure](https://daily.dev/posts/large-scale-a-b-experimentation-on-data-center-infrastructure-kasi9labf) · Google Cloud · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#ab-testing](https://daily.dev/tags/ab-testing) [View this post on daily.dev](https://daily.dev/posts/scaling-reliable-experimentation-in-a-two-sided-adtech-marketplace-zms-budget-split-segb1krmo) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Scaling Reliable Experimentation in a Two-Sided AdTech Marketplace: ZMS Budget Split","url":"https://daily.dev/posts/scaling-reliable-experimentation-in-a-two-sided-adtech-marketplace-zms-budget-split-segb1krmo","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/scaling-reliable-experimentation-in-a-two-sided-adtech-marketplace-zms-budget-split-segb1krmo"},"datePublished":"2026-09-01T12:03:47.758Z","dateModified":"2026-09-01T12:04:23.538Z","description":"Zalando's ad marketplace team describes how they solved statistical validity problems in A/B testing for their two-sided AdTech marketplace (ZMS). Because...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d742a4730ddf3e2be36b63d4ceee1fab?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d742a4730ddf3e2be36b63d4ceee1fab?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Zalando","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Zalando","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/94640378400546638ce8f6311afa73e3","url":"https://daily.dev/sources/zalando"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/scaling-reliable-experimentation-in-a-two-sided-adtech-marketplace-zms-budget-split-segb1krmo","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,ab-testing","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Zalando","item":"https://daily.dev/sources/zalando"},{"@type":"ListItem","position":3,"name":"Scaling Reliable Experimentation in a Two-Sided AdTech Marketplace: ZMS Budget Split"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/scaling-reliable-experimentation-in-a-two-sided-adtech-marketplace-zms-budget-split-segb1krmo#faq","mainEntity":[{"@type":"Question","name":"Why do standard A/B tests give biased results in a two-sided advertising marketplace with shared budgets?","acceptedAnswer":{"@type":"Answer","text":"Because control and treatment variants draw from the same finite advertiser budget, violating the Stable Unit Treatment Value Assumption (SUTVA). A more efficient variant wins more auctions and depletes the shared budget faster, starving the other variant of funding and artificially depressing its metrics, a phenomenon called cannibalization bias that dilutes the true average treatment effect. daily.dev surfaces experimentation design write-ups for teams debugging biased marketplace test results."}},{"@type":"Question","name":"What is Budget Split and how does it fix cannibalization bias in ad experiments?","acceptedAnswer":{"@type":"Answer","text":"Budget Split is a technique that partitions each ad campaign's budget into separate virtual sub-campaigns, one per experiment variant, proportional to traffic allocation. Treatment users only trigger bids against the treatment sub-budget and control users against the control sub-budget, creating two isolated marketplaces so that outperformance reflects the feature itself rather than budget cannibalization. Zalando used this to scale from 8 experiments in 2023 to over 60 by 2025. Track marketplace experimentation architecture patterns like this on daily.dev when designing your own test isolation."}},{"@type":"Question","name":"How do you run multiple concurrent A/B experiments in the same ad marketplace without them interfering?","acceptedAnswer":{"@type":"Answer","text":"Orthogonal Concurrency splits a campaign's budget into orthogonal buckets for each combination of variants across simultaneous experiments. For two concurrent experiments with A/B variants each, the budget is divided into four buckets (1A2A, 1A2B, 1B2A, 1B2B), preventing experiments from fighting over the same traffic and budget and enabling reliable concurrent testing. daily.dev collects engineering deep dives like this for teams scaling concurrent experimentation platforms."}}]} ```