--- title: A/B test models in production | daily.dev description: Together AI's platform now supports A/B testing at the endpoint level for LLM deployments in production, letting teams split live traffic between a control and... canonical: https://daily.dev/posts/a-b-test-models-in-production-w0dbvgkzs twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: A/B test models in production | daily.dev og:description: Together AI's platform now supports A/B testing at the endpoint level for LLM deployments in production, letting teams split live traffic between a control and... og:url: https://daily.dev/posts/a-b-test-models-in-production-w0dbvgkzs og:image: https://api.daily.dev/og/posts/W0DbVgkZS.png og:image:alt: A/B test models in production og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # A/B test models in production **[Together AI](https://daily.dev/sources/togetherai)** · 9 min read · 0 upvotes · 0 comments ## Summary Together AI's platform now supports A/B testing at the endpoint level for LLM deployments in production, letting teams split live traffic between a control and up to 20 variant models without embedding routing logic in application code. Unlike shadow traffic, which only validates operational soundness (latency, errors, throughput) without user exposure, this feature routes real users to variants so quality metrics like retention and task completion can be measured. Experiments are configured with fixed percentage splits that subdivide the control's share of base traffic, independent of replica counts and autoscaling. Ramping is done by resending the full member list (guarded by etags to prevent overwrites), and ending an experiment is a two-step promote-then-delete process that leaves no leftover branching logic. The post walks through a live 95/5 to 80/20 to 50/50 experiment, covering edge cases like variant degradation, sticky cohorts via sampling keys, and independent autoscaling per cohort. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How is A/B testing different from shadow traffic for evaluating a new LLM in production? Shadow traffic only proves a candidate model is operationally sound in terms of latency, errors, and throughput, since its responses are discarded and no user acts on them. A/B testing routes real users to the candidate so you can measure quality signals like retention, thumbs-up rate, and task completion. Only real exposure through an A/B split answers whether a new model is actually better for users, not just on a benchmark. _daily.dev surfaces practical infra patterns for teams deciding how to safely validate new models in production._ ### How does Together AI's endpoint-level A/B testing route traffic between a control and variant model? An A/B experiment attaches to an endpoint with exactly one control and up to 20 variants, each with a percent setting summing to 100. Requests first resolve through the base traffic split; when the control wins that split, the request is re-sampled among the experiment's arms by their percents, so the experiment subdivides the control's share rather than adding new traffic. Variant deployments must carry zero weight in the base split so their percentage is their entire traffic share. _engineers building model rollout pipelines can track infra design patterns like this via daily.dev._ ### Do A/B experiment traffic percentages change as replicas autoscale? No, A/B experiment percents are true fixed traffic shares summing to 100% and are independent of replica counts, unlike traffic-split weights which are per-ready-replica and follow capacity. This is a deliberate design choice so the split stays constant as a measurement instrument while autoscaling happens independently per cohort, with each deployment scaling on its own policy sized by its own share of traffic. _daily.dev helps engineers weighing autoscaling and experimentation trade-offs stay current on infra design choices._ ## Similar posts on daily.dev - [Configuring Dedicated Model Inference](https://daily.dev/posts/configuring-dedicated-model-inference-ux6jic7fr) · Together AI · 0 upvotes · 0 comments - [Scaling Reliable Experimentation in a Two-Sided AdTech Marketplace: ZMS Budget Split](https://daily.dev/posts/scaling-reliable-experimentation-in-a-two-sided-adtech-marketplace-zms-budget-split-segb1krmo) · Zalando · 0 upvotes · 0 comments - [Large scale A/B experimentation on data center infrastructure](https://daily.dev/posts/large-scale-a-b-experimentation-on-data-center-infrastructure-kasi9labf) · Google Cloud · 0 upvotes · 0 comments --- Tags: [#mlops](https://daily.dev/tags/mlops), [#ai-inference](https://daily.dev/tags/ai-inference), [#ab-testing](https://daily.dev/tags/ab-testing) [View this post on daily.dev](https://daily.dev/posts/a-b-test-models-in-production-w0dbvgkzs) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"A/B test models in production","url":"https://daily.dev/posts/a-b-test-models-in-production-w0dbvgkzs","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/a-b-test-models-in-production-w0dbvgkzs"},"datePublished":"2026-08-18T03:51:33.969Z","dateModified":"2026-08-18T03:52:01.821Z","description":"Together AI's platform now supports A/B testing at the endpoint level for LLM deployments in production, letting teams split live traffic between a control and...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3f0a03ac11e2b2ad417b2fb880b9277b?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3f0a03ac11e2b2ad417b2fb880b9277b?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Together AI","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Together AI","logo":"https://media.daily.dev/image/upload/s--tCjWcJfJ--/f_auto,q_auto/v1780213200/logos/togetherai?_a=BAMAMiWQ0","url":"https://daily.dev/sources/togetherai"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/a-b-test-models-in-production-w0dbvgkzs","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"mlops,ai-inference,ab-testing","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Together AI","item":"https://daily.dev/sources/togetherai"},{"@type":"ListItem","position":3,"name":"A/B test models in production"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/a-b-test-models-in-production-w0dbvgkzs#faq","mainEntity":[{"@type":"Question","name":"How is A/B testing different from shadow traffic for evaluating a new LLM in production?","acceptedAnswer":{"@type":"Answer","text":"Shadow traffic only proves a candidate model is operationally sound in terms of latency, errors, and throughput, since its responses are discarded and no user acts on them. A/B testing routes real users to the candidate so you can measure quality signals like retention, thumbs-up rate, and task completion. Only real exposure through an A/B split answers whether a new model is actually better for users, not just on a benchmark. daily.dev surfaces practical infra patterns for teams deciding how to safely validate new models in production."}},{"@type":"Question","name":"How does Together AI's endpoint-level A/B testing route traffic between a control and variant model?","acceptedAnswer":{"@type":"Answer","text":"An A/B experiment attaches to an endpoint with exactly one control and up to 20 variants, each with a percent setting summing to 100. Requests first resolve through the base traffic split; when the control wins that split, the request is re-sampled among the experiment's arms by their percents, so the experiment subdivides the control's share rather than adding new traffic. Variant deployments must carry zero weight in the base split so their percentage is their entire traffic share. engineers building model rollout pipelines can track infra design patterns like this via daily.dev."}},{"@type":"Question","name":"Do A/B experiment traffic percentages change as replicas autoscale?","acceptedAnswer":{"@type":"Answer","text":"No, A/B experiment percents are true fixed traffic shares summing to 100% and are independent of replica counts, unlike traffic-split weights which are per-ready-replica and follow capacity. This is a deliberate design choice so the split stays constant as a measurement instrument while autoscaling happens independently per cohort, with each deployment scaling on its own policy sized by its own share of traffic. daily.dev helps engineers weighing autoscaling and experimentation trade-offs stay current on infra design choices."}}]} ```