--- title: A/B test models in production | daily.dev description: Together AI's platform now supports A/B testing at the endpoint level for LLM deployments in production, letting teams split live traffic between a control and... canonical: https://daily.dev/posts/a-b-test-models-in-production-w0dbvgkzs twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: A/B test models in production | daily.dev og:description: Together AI's platform now supports A/B testing at the endpoint level for LLM deployments in production, letting teams split live traffic between a control and... og:url: https://daily.dev/posts/a-b-test-models-in-production-w0dbvgkzs og:image: https://api.daily.dev/og/posts/W0DbVgkZS.png og:image:alt: A/B test models in production og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # A/B test models in production **[Together AI](https://daily.dev/sources/togetherai)** · 9 min read · 0 upvotes · 0 comments ## Summary Together AI's platform now supports A/B testing at the endpoint level for LLM deployments in production, letting teams split live traffic between a control and up to 20 variant models without embedding routing logic in application code. Unlike shadow traffic, which only validates operational soundness (latency, errors, throughput) without user exposure, this feature routes real users to variants so quality metrics like retention and task completion can be measured. Experiments are configured with fixed percentage splits that subdivide the control's share of base traffic, independent of replica counts and autoscaling. Ramping is done by resending the full member list (guarded by etags to prevent overwrites), and ending an experiment is a two-step promote-then-delete process that leaves no leftover branching logic. The post walks through a live 95/5 to 80/20 to 50/50 experiment, covering edge cases like variant degradation, sticky cohorts via sampling keys, and independent autoscaling per cohort. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How is A/B testing for LLM models different from shadow traffic testing? Shadow traffic only proves a candidate model is operationally sound in terms of latency, errors, and throughput, since its responses are discarded and no user ever acts on them. A/B testing routes a real percentage of live traffic to the candidate so actual user outcomes like retention, thumbs-up rate, or task completion can be measured, which shadow traffic cannot reveal. _Teams evaluating model rollout strategies can follow ongoing coverage of production LLM testing patterns on daily.dev._ ### How do I keep A/B test traffic percentages accurate when autoscaling changes replica counts? A/B experiment percentages are true fixed traffic shares that sum to 100% and are set deliberately independent of replica counts, unlike traffic-split weights which are per-ready-replica and follow capacity. This keeps the measurement split constant while data is collected, even as deployments scale up or down during the test. _Engineers designing reliable model experiments can track infrastructure patterns like this on daily.dev._ ### How do I ensure a user stays in the same A/B test group across multiple requests? Cohort assignment uses the request's sampling key, such as a top-level prompt_cache_key or user field in the request body, so requests carrying the same key route consistently to the same arm. Requests without a key are assigned effectively at random per request, so a stable user field should be sent when per-user consistency matters, especially for multi-turn quality comparisons. _Developers building sticky experiment cohorts for multi-turn LLM apps can follow related guidance on daily.dev._ ## Similar posts on daily.dev - [Configuring Dedicated Model Inference](https://daily.dev/posts/configuring-dedicated-model-inference-ux6jic7fr) · Together AI · 0 upvotes · 0 comments - [Inside Google’s System for Coordinated A/B Testing Across Its Global Service Fleet](https://daily.dev/posts/inside-google-s-system-for-coordinated-a-b-testing-across-its-global-service-fleet-dcjl4coz4) · InfoQ · 0 upvotes · 0 comments - [Scaling Reliable Experimentation in a Two-Sided AdTech Marketplace: ZMS Budget Split](https://daily.dev/posts/scaling-reliable-experimentation-in-a-two-sided-adtech-marketplace-zms-budget-split-segb1krmo) · Zalando · 0 upvotes · 0 comments - [Large scale A/B experimentation on data center infrastructure](https://daily.dev/posts/large-scale-a-b-experimentation-on-data-center-infrastructure-kasi9labf) · Google Cloud · 0 upvotes · 0 comments --- Tags: [#ab-testing](https://daily.dev/tags/ab-testing), [#ai-inference](https://daily.dev/tags/ai-inference), [#mlops](https://daily.dev/tags/mlops), [#together-ai](https://daily.dev/tags/together-ai) [View this post on daily.dev](https://daily.dev/posts/a-b-test-models-in-production-w0dbvgkzs) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"A/B test models in production","url":"https://daily.dev/posts/a-b-test-models-in-production-w0dbvgkzs","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/a-b-test-models-in-production-w0dbvgkzs"},"datePublished":"2026-08-18T03:51:33.969Z","dateModified":"2026-10-15T02:22:39.925Z","description":"Together AI's platform now supports A/B testing at the endpoint level for LLM deployments in production, letting teams split live traffic between a control and...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3f0a03ac11e2b2ad417b2fb880b9277b?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3f0a03ac11e2b2ad417b2fb880b9277b?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Together AI","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Together AI","logo":"https://media.daily.dev/image/upload/s--tCjWcJfJ--/f_auto,q_auto/v1780213200/logos/togetherai?_a=BAMAMiWQ0","url":"https://daily.dev/sources/togetherai"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/a-b-test-models-in-production-w0dbvgkzs","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ab-testing,ai-inference,mlops,together-ai","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Together AI","item":"https://daily.dev/sources/togetherai"},{"@type":"ListItem","position":3,"name":"A/B test models in production"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/a-b-test-models-in-production-w0dbvgkzs#faq","mainEntity":[{"@type":"Question","name":"How is A/B testing for LLM models different from shadow traffic testing?","acceptedAnswer":{"@type":"Answer","text":"Shadow traffic only proves a candidate model is operationally sound in terms of latency, errors, and throughput, since its responses are discarded and no user ever acts on them. A/B testing routes a real percentage of live traffic to the candidate so actual user outcomes like retention, thumbs-up rate, or task completion can be measured, which shadow traffic cannot reveal. Teams evaluating model rollout strategies can follow ongoing coverage of production LLM testing patterns on daily.dev."}},{"@type":"Question","name":"How do I keep A/B test traffic percentages accurate when autoscaling changes replica counts?","acceptedAnswer":{"@type":"Answer","text":"A/B experiment percentages are true fixed traffic shares that sum to 100% and are set deliberately independent of replica counts, unlike traffic-split weights which are per-ready-replica and follow capacity. This keeps the measurement split constant while data is collected, even as deployments scale up or down during the test. Engineers designing reliable model experiments can track infrastructure patterns like this on daily.dev."}},{"@type":"Question","name":"How do I ensure a user stays in the same A/B test group across multiple requests?","acceptedAnswer":{"@type":"Answer","text":"Cohort assignment uses the request's sampling key, such as a top-level prompt_cache_key or user field in the request body, so requests carrying the same key route consistently to the same arm. Requests without a key are assigned effectively at random per request, so a stable user field should be sent when per-user consistency matters, especially for multi-turn quality comparisons. Developers building sticky experiment cohorts for multi-turn LLM apps can follow related guidance on daily.dev."}}]} ```