--- title: What We Miss About Missing Values | daily.dev description: Missing data is framed not as a nuisance to be dropped or imputed away, but as a signal generated by the same process that produces observed data. Rubin's... canonical: https://daily.dev/posts/what-we-miss-about-missing-values-hkvdfcwes twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: What We Miss About Missing Values | daily.dev og:description: Missing data is framed not as a nuisance to be dropped or imputed away, but as a signal generated by the same process that produces observed data. Rubin's... og:url: https://daily.dev/posts/what-we-miss-about-missing-values-hkvdfcwes og:image: https://api.daily.dev/og/posts/hKVdFCWes.png og:image:alt: What We Miss About Missing Values og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # What We Miss About Missing Values **[Towards Data Science](https://daily.dev/sources/tds)** · 13 min read · 0 upvotes · 0 comments ## Summary Missing data is framed not as a nuisance to be dropped or imputed away, but as a signal generated by the same process that produces observed data. Rubin's classic taxonomy (MCAR, MAR, MNAR) sets the terms for how to reason about gaps, with Heckman's selection models, inverse probability weighting/AIPW, and multiple imputation offered as methods that encode different assumptions about why data are missing. Sensitivity analysis (tipping-point methods) is presented for cases where the mechanism can't be identified from data alone. The piece also distinguishes inference from prediction: in prediction tasks, missingness itself can be informative (via the missing indicator method or tree-based models' native handling), while in inference it's a bias source to correct for. The ICH E9(R1) estimand framework is invoked to argue that how missingness should be handled depends on the precise question being asked. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is the difference between data missing at random (MAR) and missing not at random (MNAR)? Data are missing at random when the probability of a gap depends only on values that are observed, so unbiased estimation is possible by conditioning on those observed variables. Data are missing not at random when the missingness probability depends on the unobserved value itself, meaning no method conditioning only on observed data can identify the missing distribution without extra assumptions. The distinction can't be tested from the data alone; it must be argued from domain knowledge. _Understanding these missing-data assumptions helps data scientists judge whether an analysis pipeline on daily.dev-shared research is trustworthy._ ### How does the missing indicator method improve predictive models when data is missing? The missing indicator method imputes each missing value while adding a separate binary feature flagging whether that value was originally absent, letting a model learn from both the imputed value and the fact of its absence. Research by Van Ness et al. shows this can boost predictive performance when missingness itself is informative, though too many indicators in high-dimensional data can cause overfitting. It only helps if the same missingness pattern exists at prediction time as during training. _Engineers comparing imputation strategies for production ML pipelines can track this kind of method guidance on daily.dev._ ### How does Heckman's selection model correct for sample selection bias? Heckman's approach pairs an outcome equation with a selection equation describing whether the outcome is observed at all, allowing unobserved factors driving selection and the outcome to be correlated instead of assumed independent. The original two-step estimator models the observation probability first, then corrects the outcome equation using the inverse Mills ratio, which captures the expected bias from only observing outcomes past a selection threshold. It requires assuming the unobserved components follow a joint normal distribution. _Data scientists weighing selection-model assumptions against reweighting methods can follow methodological deep-dives like this on daily.dev._ ## Similar posts on daily.dev - [Why a Missing Value Can Be More Valuable Than the Value Itself](https://daily.dev/posts/why-a-missing-value-can-be-more-valuable-than-the-value-itself-vm1rdzqwg) · Medium · 0 upvotes · 0 comments - [Why you should not use mean imputation for missing data](https://daily.dev/posts/why-you-should-not-use-mean-imputation-for-missing-data-u31ygbga1) · R-bloggers · 0 upvotes · 0 comments - [Why MissForest Fails in Prediction Tasks: A Key Limitation You Need to Keep in Mind](https://daily.dev/posts/why-missforest-fails-in-prediction-tasks-a-key-limitation-you-need-to-keep-in-mind-xprvgemlz) · Towards Data Science · 0 upvotes · 0 comments - [A practical introduction to multiple imputation of missing data with the R-package mice workshop](https://daily.dev/posts/a-practical-introduction-to-multiple-imputation-of-missing-data-with-the-r-package-mice-workshop-t4t617dlk) · R-bloggers · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#data-science](https://daily.dev/tags/data-science), [#statistics](https://daily.dev/tags/statistics) [View this post on daily.dev](https://daily.dev/posts/what-we-miss-about-missing-values-hkvdfcwes) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"What We Miss About Missing Values","url":"https://daily.dev/posts/what-we-miss-about-missing-values-hkvdfcwes","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/what-we-miss-about-missing-values-hkvdfcwes"},"datePublished":"2026-09-01T14:48:03.261Z","dateModified":"2026-09-01T14:48:29.414Z","description":"Missing data is framed not as a nuisance to be dropped or imputed away, but as a signal generated by the same process that produces observed data. Rubin's...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/230aa88f1c10546610cce05603875f84?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/230aa88f1c10546610cce05603875f84?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Towards Data Science","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Towards Data Science","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/tds","url":"https://daily.dev/sources/tds"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/what-we-miss-about-missing-values-hkvdfcwes","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,data-science,statistics","timeRequired":"PT13M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Towards Data Science","item":"https://daily.dev/sources/tds"},{"@type":"ListItem","position":3,"name":"What We Miss About Missing Values"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/what-we-miss-about-missing-values-hkvdfcwes#faq","mainEntity":[{"@type":"Question","name":"What is the difference between data missing at random (MAR) and missing not at random (MNAR)?","acceptedAnswer":{"@type":"Answer","text":"Data are missing at random when the probability of a gap depends only on values that are observed, so unbiased estimation is possible by conditioning on those observed variables. Data are missing not at random when the missingness probability depends on the unobserved value itself, meaning no method conditioning only on observed data can identify the missing distribution without extra assumptions. The distinction can't be tested from the data alone; it must be argued from domain knowledge. Understanding these missing-data assumptions helps data scientists judge whether an analysis pipeline on daily.dev-shared research is trustworthy."}},{"@type":"Question","name":"How does the missing indicator method improve predictive models when data is missing?","acceptedAnswer":{"@type":"Answer","text":"The missing indicator method imputes each missing value while adding a separate binary feature flagging whether that value was originally absent, letting a model learn from both the imputed value and the fact of its absence. Research by Van Ness et al. shows this can boost predictive performance when missingness itself is informative, though too many indicators in high-dimensional data can cause overfitting. It only helps if the same missingness pattern exists at prediction time as during training. Engineers comparing imputation strategies for production ML pipelines can track this kind of method guidance on daily.dev."}},{"@type":"Question","name":"How does Heckman's selection model correct for sample selection bias?","acceptedAnswer":{"@type":"Answer","text":"Heckman's approach pairs an outcome equation with a selection equation describing whether the outcome is observed at all, allowing unobserved factors driving selection and the outcome to be correlated instead of assumed independent. The original two-step estimator models the observation probability first, then corrects the outcome equation using the inverse Mills ratio, which captures the expected bias from only observing outcomes past a selection threshold. It requires assuming the unobserved components follow a joint normal distribution. Data scientists weighing selection-model assumptions against reweighting methods can follow methodological deep-dives like this on daily.dev."}}]} ```