--- title: Product Experimentation with Doubly Robust Estimation:... description: Doubly robust estimation via the Augmented Inverse-Probability Weighting (AIPW) estimator is a causal inference technique that remains consistent if either the... canonical: https://daily.dev/posts/product-experimentation-with-doubly-robust-estimation-when-both-your-models-are-wrong-in-llm-applic-vxma9z7do twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Product Experimentation with Doubly Robust Estimation: When Both Your Models Are Wrong in LLM Applications | daily.dev og:description: Doubly robust estimation via the Augmented Inverse-Probability Weighting (AIPW) estimator is a causal inference technique that remains consistent if either the... og:url: https://daily.dev/posts/product-experimentation-with-doubly-robust-estimation-when-both-your-models-are-wrong-in-llm-applic-vxma9z7do og:image: https://api.daily.dev/og/posts/VxMA9z7Do.png og:image:alt: Product Experimentation with Doubly Robust Estimation: When Both Your Models Are Wrong in LLM Applications og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Product Experimentation with Doubly Robust Estimation: When Both Your Models Are Wrong in LLM Applications **[freeCodeCamp](https://daily.dev/sources/freecodecamp)** · 22 min read · 1 upvotes · 0 comments ## Summary Doubly robust estimation via the Augmented Inverse-Probability Weighting (AIPW) estimator is a causal inference technique that remains consistent if either the propensity model or the outcome model is correctly specified — both must fail simultaneously for the estimator to break. The tutorial walks through implementing AIPW from scratch using scikit-learn on a 50,000-user synthetic LLM opt-in dataset where the ground-truth treatment effect is +8 percentage points. A naïve comparison inflates the estimate to +21 percentage points due to selection bias. Step-by-step code covers fitting the propensity model, fitting separate outcome models for treated and control groups, combining them into the AIPW formula, adding bootstrap confidence intervals, and empirically proving double robustness by deliberately misspecifying one model at a time. Limitations covered include unmeasured confounding, extreme propensity score variance inflation, finite-sample inefficiency, and the need for cross-fitting when using flexible ML models in production. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is the AIPW (augmented inverse-probability weighting) formula for estimating average treatment effect? The AIPW estimator combines regression adjustment and inverse-probability weighting into one formula: ATE = mean(m1(X) - m0(X) + T*(Y - m1(X))/e(X) - (1-T)*(Y - m0(X))/(1-e(X))), where e(X) is the propensity score, m1(X)/m0(X) are predicted outcomes under treatment/control, T is the treatment indicator, and Y is the observed outcome. It stays consistent if either the propensity or outcome model is correctly specified. _daily.dev surfaces causal-inference techniques like this for teams validating product experiment results._ ### Why is doubly robust estimation better than propensity weighting or regression adjustment alone for analyzing opt-in AI feature experiments? Doubly robust estimation (AIPW) remains a consistent estimate of the treatment effect if either the propensity model or the outcome model is correctly specified, whereas propensity weighting alone fails if the propensity model is wrong and regression adjustment alone fails if the outcome model is wrong. Both models must fail simultaneously for AIPW to break, unlike single-model methods that bet everything on one specification being correct. _Developers choosing between causal inference methods can track these trade-offs on daily.dev._ ### What happens to the AIPW estimate when the propensity model is deliberately misspecified to a constant value? When propensity scores are replaced with a constant 0.3 for every user, pure IPW breaks and reproduces the naive biased estimate (around +0.21 in a test dataset), but AIPW stays near the true effect (+0.0847) because the correctly specified outcome models carry the estimation forward and the IPW correction noise averages out across the sample. _Teams validating experiment pipelines can follow deep dives like this on daily.dev before shipping causal analyses._ ## Similar posts on daily.dev - [Trusting the Untestable: Validation and Diagnostics for the Doubly Robust Models](https://daily.dev/posts/trusting-the-untestable-validation-and-diagnostics-for-the-doubly-robust-models-r18xv6fnc) · Lyft Engineering · 0 upvotes · 0 comments - [Product Experimentation with Propensity Scores: Causal Inference for LLM-Based Features in Python](https://daily.dev/posts/product-experimentation-with-propensity-scores-causal-inference-for-llm-based-features-in-python-uju401omp) · freeCodeCamp · 4 upvotes · 0 comments - [Stop Blaming the Data: A Better Way to Handle Covariance Shift](https://daily.dev/posts/stop-blaming-the-data-a-better-way-to-handle-covariance-shift-2nbp2klza) · Towards Data Science · 3 upvotes · 0 comments - [Issue \#137 - From Randomized to Natural Experiments](https://daily.dev/posts/issue-137---from-randomized-to-natural-experiments-9lvbt7dwg) · Machine Learning Pills · 0 upvotes · 0 comments - [The Causal Inference Playbook: Advanced Methods Every Data Scientist Should Master](https://daily.dev/posts/the-causal-inference-playbook-advanced-methods-every-data-scientist-should-master-pulnpkh7h) · Towards Data Science · 1 upvotes · 0 comments --- Tags: [#python](https://daily.dev/tags/python), [#llm](https://daily.dev/tags/llm), [#scikit](https://daily.dev/tags/scikit) [View this post on daily.dev](https://daily.dev/posts/product-experimentation-with-doubly-robust-estimation-when-both-your-models-are-wrong-in-llm-applic-vxma9z7do) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Product Experimentation with Doubly Robust Estimation: When Both Your Models Are Wrong in LLM Applications","url":"https://daily.dev/posts/product-experimentation-with-doubly-robust-estimation-when-both-your-models-are-wrong-in-llm-applic-vxma9z7do","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/product-experimentation-with-doubly-robust-estimation-when-both-your-models-are-wrong-in-llm-applic-vxma9z7do"},"datePublished":"2026-08-11T04:29:14.724Z","dateModified":"2026-09-14T08:35:52.727Z","description":"Doubly robust estimation via the Augmented Inverse-Probability Weighting (AIPW) estimator is a causal inference technique that remains consistent if either the...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/299447f171755e282ada9ad8a512d647?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/299447f171755e282ada9ad8a512d647?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"freeCodeCamp","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"freeCodeCamp","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1628412854/logos/freecodecamp","url":"https://daily.dev/sources/freecodecamp"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/product-experimentation-with-doubly-robust-estimation-when-both-your-models-are-wrong-in-llm-applic-vxma9z7do","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,llm,scikit","timeRequired":"PT22M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"freeCodeCamp","item":"https://daily.dev/sources/freecodecamp"},{"@type":"ListItem","position":3,"name":"Product Experimentation with Doubly Robust Estimation: When Both Your Models Are Wrong in LLM Applications"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/product-experimentation-with-doubly-robust-estimation-when-both-your-models-are-wrong-in-llm-applic-vxma9z7do#faq","mainEntity":[{"@type":"Question","name":"What is the AIPW (augmented inverse-probability weighting) formula for estimating average treatment effect?","acceptedAnswer":{"@type":"Answer","text":"The AIPW estimator combines regression adjustment and inverse-probability weighting into one formula: ATE = mean(m1(X) - m0(X) + T*(Y - m1(X))/e(X) - (1-T)*(Y - m0(X))/(1-e(X))), where e(X) is the propensity score, m1(X)/m0(X) are predicted outcomes under treatment/control, T is the treatment indicator, and Y is the observed outcome. It stays consistent if either the propensity or outcome model is correctly specified. daily.dev surfaces causal-inference techniques like this for teams validating product experiment results."}},{"@type":"Question","name":"Why is doubly robust estimation better than propensity weighting or regression adjustment alone for analyzing opt-in AI feature experiments?","acceptedAnswer":{"@type":"Answer","text":"Doubly robust estimation (AIPW) remains a consistent estimate of the treatment effect if either the propensity model or the outcome model is correctly specified, whereas propensity weighting alone fails if the propensity model is wrong and regression adjustment alone fails if the outcome model is wrong. Both models must fail simultaneously for AIPW to break, unlike single-model methods that bet everything on one specification being correct. Developers choosing between causal inference methods can track these trade-offs on daily.dev."}},{"@type":"Question","name":"What happens to the AIPW estimate when the propensity model is deliberately misspecified to a constant value?","acceptedAnswer":{"@type":"Answer","text":"When propensity scores are replaced with a constant 0.3 for every user, pure IPW breaks and reproduces the naive biased estimate (around +0.21 in a test dataset), but AIPW stays near the true effect (+0.0847) because the correctly specified outcome models carry the estimation forward and the IPW correction noise averages out across the sample. Teams validating experiment pipelines can follow deep dives like this on daily.dev before shipping causal analyses."}}]} ```