--- title: When Can LLMs Replace Humans in A/B Tests? | daily.dev description: Spotify researchers formalize when LLM predictions can validly substitute for human data in A/B tests, using surrogate endpoint theory from biostatistics.... canonical: https://daily.dev/posts/when-can-llms-replace-humans-in-a-b-tests--joccfpevz twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: When Can LLMs Replace Humans in A/B Tests? | daily.dev og:description: Spotify researchers formalize when LLM predictions can validly substitute for human data in A/B tests, using surrogate endpoint theory from biostatistics.... og:url: https://daily.dev/posts/when-can-llms-replace-humans-in-a-b-tests--joccfpevz og:image: https://api.daily.dev/og/posts/jOCcFpEvZ.png og:image:alt: When Can LLMs Replace Humans in A/B Tests? og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # When Can LLMs Replace Humans in A/B Tests? **[Spotify Labs](https://daily.dev/sources/spotify)** · 9 min read · 6 upvotes · 0 comments ## Summary Spotify researchers formalize when LLM predictions can validly substitute for human data in A/B tests, using surrogate endpoint theory from biostatistics. Testing on the Upworthy headline dataset, raw GPT-4o-mini predictions recovered only 39% of the true treatment effect, biased systematically toward zero. Two conditions—surrogacy and comparability—must hold for calibration to work, and calibrating with flexible ML models (random forest, gradient-boosted trees) succeeded where linear OLS calibration failed a falsification test. Crucially, these conditions can't be verified for genuinely new treatments, meaning LLM-based experimentation is least trustworthy exactly where it would be most valuable. The authors conclude human experiments remain indispensable for real product innovation, though LLM predictions can still help filter weak ideas or serve as variance-reduction covariates. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Can LLM predictions replace human responses in A/B tests? Only under two strict, unverifiable-for-new-treatments conditions: surrogacy (the LLM output must fully capture everything about the treatment that affects human behavior) and comparability (the mapping between LLM predictions and human outcomes must stay stable across experiments). Testing GPT-4o-mini on the Upworthy headline dataset, raw LLM predictions recovered only 39% of the real human treatment effect, systematically underestimating impact. _Teams weighing LLM-based experimentation against traditional A/B testing can track this kind of methods research on daily.dev._ ### Why did linear calibration fail to recover the human treatment effect from LLM predictions in A/B tests? Ordinary least squares calibration was too rigid to capture the nonlinear relationship between LLM predictions and human outcomes, landing 3.8 standard errors from the human benchmark on the Upworthy dataset. Flexible machine learning models, specifically random forests and gradient-boosted trees, calibrated the predictions well enough that the resulting estimate fell within the human effect's sampling error. _Practitioners choosing calibration methods for proxy metrics can follow this kind of experimentation research on daily.dev._ ### Why is LLM-based A/B testing least reliable for genuinely new product treatments? The surrogacy and comparability conditions needed to trust LLM predictions can only be partially checked against historical data, and they get weaker the further a new treatment departs from what has already been tested. For a brand-new UI paradigm, pricing model, or unprecedented feature, these assumptions are inherently untestable, so LLMs offer the least reliable substitute exactly where replacing human experiments would be most valuable. _Anyone deciding when to trust AI-driven experimentation over real user tests can follow this reasoning on daily.dev._ ## Similar posts on daily.dev - [Can LLMs Replace Survey Respondents?](https://daily.dev/posts/can-llms-replace-survey-respondents--ak7redkzp) · Towards Data Science · 0 upvotes · 0 comments - [From IR to RecSys: Evaluating LLM-based Judges in Cranfield-style Recommendation Collections](https://daily.dev/posts/from-ir-to-recsys-evaluating-llm-based-judges-in-cranfield-style-recommendation-collections-kilhqnwnm) · Spotify Research · 0 upvotes · 0 comments --- Tags: [#data-science](https://daily.dev/tags/data-science), [#llm](https://daily.dev/tags/llm), [#statistics](https://daily.dev/tags/statistics), [#ab-testing](https://daily.dev/tags/ab-testing) [View this post on daily.dev](https://daily.dev/posts/when-can-llms-replace-humans-in-a-b-tests--joccfpevz) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"When Can LLMs Replace Humans in A/B Tests?","url":"https://daily.dev/posts/when-can-llms-replace-humans-in-a-b-tests--joccfpevz","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/when-can-llms-replace-humans-in-a-b-tests--joccfpevz"},"datePublished":"2026-08-13T19:18:21.071Z","dateModified":"2026-09-14T08:49:34.035Z","description":"Spotify researchers formalize when LLM predictions can validly substitute for human data in A/B tests, using surrogate endpoint theory from biostatistics....","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/fec706b2942b2851399d6363a57b5d17?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/fec706b2942b2851399d6363a57b5d17?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Spotify Labs","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Spotify Labs","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/spotify","url":"https://daily.dev/sources/spotify"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/when-can-llms-replace-humans-in-a-b-tests--joccfpevz","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":6},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-science,llm,statistics,ab-testing","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Spotify Labs","item":"https://daily.dev/sources/spotify"},{"@type":"ListItem","position":3,"name":"When Can LLMs Replace Humans in A/B Tests?"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/when-can-llms-replace-humans-in-a-b-tests--joccfpevz#faq","mainEntity":[{"@type":"Question","name":"Can LLM predictions replace human responses in A/B tests?","acceptedAnswer":{"@type":"Answer","text":"Only under two strict, unverifiable-for-new-treatments conditions: surrogacy (the LLM output must fully capture everything about the treatment that affects human behavior) and comparability (the mapping between LLM predictions and human outcomes must stay stable across experiments). Testing GPT-4o-mini on the Upworthy headline dataset, raw LLM predictions recovered only 39% of the real human treatment effect, systematically underestimating impact. Teams weighing LLM-based experimentation against traditional A/B testing can track this kind of methods research on daily.dev."}},{"@type":"Question","name":"Why did linear calibration fail to recover the human treatment effect from LLM predictions in A/B tests?","acceptedAnswer":{"@type":"Answer","text":"Ordinary least squares calibration was too rigid to capture the nonlinear relationship between LLM predictions and human outcomes, landing 3.8 standard errors from the human benchmark on the Upworthy dataset. Flexible machine learning models, specifically random forests and gradient-boosted trees, calibrated the predictions well enough that the resulting estimate fell within the human effect's sampling error. Practitioners choosing calibration methods for proxy metrics can follow this kind of experimentation research on daily.dev."}},{"@type":"Question","name":"Why is LLM-based A/B testing least reliable for genuinely new product treatments?","acceptedAnswer":{"@type":"Answer","text":"The surrogacy and comparability conditions needed to trust LLM predictions can only be partially checked against historical data, and they get weaker the further a new treatment departs from what has already been tested. For a brand-new UI paradigm, pricing model, or unprecedented feature, these assumptions are inherently untestable, so LLMs offer the least reliable substitute exactly where replacing human experiments would be most valuable. Anyone deciding when to trust AI-driven experimentation over real user tests can follow this reasoning on daily.dev."}}]} ```