Spotify researchers formalize when LLM predictions can validly substitute for human data in A/B tests, using surrogate endpoint theory from biostatistics. Testing on the Upworthy headline dataset, raw GPT-4o-mini predictions recovered only 39% of the true treatment effect, biased systematically toward zero. Two conditions—surrogacy and comparability—must hold for calibration to work, and calibrating with flexible ML models (random forest, gradient-boosted trees) succeeded where linear OLS calibration failed a falsification test. Crucially, these conditions can't be verified for genuinely new treatments, meaning LLM-based experimentation is least trustworthy exactly where it would be most valuable. The authors conclude human experiments remain indispensable for real product innovation, though LLM predictions can still help filter weak ideas or serve as variance-reduction covariates.
Table of contents
Raw LLM predictions are biased, not just noisyTwo conditions make LLM outputs valid surrogatesCalibration recovers the effect, but only with the right methodThe real limitation is about future interventionsCalibration needs the data you are trying to avoid collectingFinal words: User experiments work by design, LLM-based experiments by assumptionQuestions this post answers
Can I use LLM predictions instead of real user data to run A/B tests?
Only under two strong, often unverifiable assumptions: surrogacy (the LLM output fully captures everything about the treatment that affects human behavior) and comparability (the mapping between LLM predictions and human outcomes stays stable across experiments). Raw GPT-4o-mini predictions on the Upworthy headline dataset recovered just 39% of the true human treatment effect without calibration, showing uncalibrated LLM outputs are systematically biased toward zero, not merely noisy. Anyone weighing LLM-based experimentation against real user testing can track this kind of research on daily.dev.
Which calibration method works best for mapping LLM predictions to human A/B test outcomes?
Machine learning calibration methods, specifically random forest and gradient-boosted trees, worked better than linear calibration using ordinary least squares (OLS) on the Upworthy headline dataset. OLS calibration failed a falsification test, landing 3.8 standard errors from the human benchmark, because it was too rigid to capture the nonlinear relationship between LLM predictions and human behavior; the ML-based methods fell within human sampling error. Data scientists comparing calibration approaches for LLM surrogates can follow this kind of experimentation research on daily.dev.
Why can't LLM-based A/B testing fully replace human experiments for new product features?
Because the surrogacy and comparability assumptions needed for LLM predictions to substitute for human outcomes can only be checked against historical data and can never be proven for a treatment that has never been tested before. The further a new treatment departs from past experiments, such as a new UI paradigm or pricing model, the less trustworthy the LLM stand-in becomes, so human experiments remain necessary for genuine product innovation. Teams weighing faster LLM-based testing against reliable human experiments can follow this debate on daily.dev.