--- title: Why Great Models Fail: Lessons From 9 Years of Deploying... description: A machine learning consultant with nearly a decade of experience shares lessons from deploying ML models in industry, centered on why technically strong models... canonical: https://daily.dev/posts/why-great-models-fail-lessons-from-9-years-of-deploying-ml-models---megan-robertson-742qhwjzr twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Why Great Models Fail: Lessons From 9 Years of Deploying ML Models - Megan Robertson | daily.dev og:description: A machine learning consultant with nearly a decade of experience shares lessons from deploying ML models in industry, centered on why technically strong models... og:url: https://daily.dev/posts/why-great-models-fail-lessons-from-9-years-of-deploying-ml-models---megan-robertson-742qhwjzr og:image: https://api.daily.dev/og/posts/742qhwjZR.png og:image:alt: Why Great Models Fail: Lessons From 9 Years of Deploying ML Models - Megan Robertson og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Why Great Models Fail: Lessons From 9 Years of Deploying ML Models - Megan Robertson **[NDC Conferences](https://daily.dev/sources/ndc)** · 59 min read · 0 upvotes · 0 comments ## Summary A machine learning consultant with nearly a decade of experience shares lessons from deploying ML models in industry, centered on why technically strong models often fail to reach or survive in production. She walks through a personal failure case (a social media topic-modeling project) that performed well quantitatively but lacked stakeholder buy-in and hit unexpected data costs. The talk covers a framework for scoping ML projects (defining stakeholders, problems, and success metrics; understanding constraints and risks; identifying solutions via MVP; planning maintenance), plus a breakdown of model drift types (concept, data, label, reality drift) and mitigation strategies like feature re-engineering, retraining, algorithm changes, or retiring a model. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is model drift and what are the different types of it? Model drift is the phenomenon where a machine learning model's performance degrades over time because the world it was trained on changes. There are four types: concept drift (the relationship between input and target variables changes, as in evolving credit card fraud patterns), data drift (input feature distributions change, e.g. seasonal products), label drift (the target variable's distribution shifts, e.g. a sudden spike in churn), and reality drift (a sudden, drastic real-world shift with no prior signal in the data, like COVID-19). _Track evolving strategies for detecting and handling model drift as production ML practices change on daily.dev._ ### What are common reasons a well-performing machine learning model fails to make it into production? Two dominant reasons are poor project scoping and the fact that the world changes over time. Poor scoping includes building without consulting stakeholders, evaluating success on vibes instead of quantitative KPIs, ignoring budget and compute costs, and delivering an overcomplicated solution when a simpler model performs nearly as well. Even a mathematically strong model can end up unused if its value doesn't clearly exceed its cost and risk. _Compare real-world lessons on shipping ML projects successfully as new case studies surface on daily.dev._ ### How can I detect and fix data drift or model performance degradation in a production ML pipeline? Detection relies on monitoring summary statistics and statistical tests like the chi-square and Kolmogorov-Smirnov tests to compare current data distributions against historical baselines, alongside tracking model metrics such as MAE, MSE, precision, recall, F1, or semantic similarity for language models. Fixes include redoing feature engineering, retraining on more recent data, switching to a different algorithm, or retiring the model entirely if it no longer serves its purpose. _Follow practical monitoring and retraining strategies for production ML systems on daily.dev._ ## Similar posts on daily.dev - [Why Most Machine Learning Projects Fail to Reach Production](https://daily.dev/posts/why-most-machine-learning-projects-fail-to-reach-production-opcx6gdce) · InfoQ · 0 upvotes · 0 comments - [My Models Failed. That’s How I Became a Better Data Scientist.](https://daily.dev/posts/my-models-failed-that-s-how-i-became-a-better-data-scientist--qxnalkqmb) · Towards Data Science · 0 upvotes · 0 comments - [Why Enterprise AI Projects Fail: 5 Root Causes & Fixes](https://daily.dev/posts/why-enterprise-ai-projects-fail-5-root-causes-fixes-roxbg76ej) · Netguru · 0 upvotes · 0 comments - [Why Your ML Model Works in Training But Fails in Production](https://daily.dev/posts/why-your-ml-model-works-in-training-but-fails-in-production-ugggi6vv8) · Towards Data Science · 1 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#nlp](https://daily.dev/tags/nlp), [#mlops](https://daily.dev/tags/mlops) [View this post on daily.dev](https://daily.dev/posts/why-great-models-fail-lessons-from-9-years-of-deploying-ml-models---megan-robertson-742qhwjzr) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Why Great Models Fail: Lessons From 9 Years of Deploying ML Models - Megan Robertson","url":"https://daily.dev/posts/why-great-models-fail-lessons-from-9-years-of-deploying-ml-models---megan-robertson-742qhwjzr","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/why-great-models-fail-lessons-from-9-years-of-deploying-ml-models---megan-robertson-742qhwjzr"},"datePublished":"2026-08-13T12:07:23.730Z","dateModified":"2026-08-13T12:07:50.261Z","description":"A machine learning consultant with nearly a decade of experience shares lessons from deploying ML models in industry, centered on why technically strong models...","image":"https://i.ytimg.com/vi/WKQFUmn-WSc/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/WKQFUmn-WSc/sddefault.jpg","isAccessibleForFree":true,"articleSection":"NDC Conferences","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"NDC Conferences","logo":"https://media.daily.dev/image/upload/s--SiIyojEj--/f_auto/v1748760369/logos/ndc","url":"https://daily.dev/sources/ndc"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/why-great-models-fail-lessons-from-9-years-of-deploying-ml-models---megan-robertson-742qhwjzr","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,nlp,mlops","timeRequired":"PT59M","video":{"@type":"VideoObject","name":"Why Great Models Fail: Lessons From 9 Years of Deploying ML Models - Megan Robertson","description":"A machine learning consultant with nearly a decade of experience shares lessons from deploying ML models in industry, centered on why technically strong models...","thumbnailUrl":"https://i.ytimg.com/vi/WKQFUmn-WSc/sddefault.jpg","uploadDate":"2026-08-13T12:07:23.730Z","duration":"PT59M","url":"https://api.daily.dev/r/742qhwjZR","embedUrl":"https://www.youtube.com/embed/WKQFUmn-WSc"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"NDC Conferences","item":"https://daily.dev/sources/ndc"},{"@type":"ListItem","position":3,"name":"Why Great Models Fail: Lessons From 9 Years of Deploying ML Models - Megan Robertson"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/why-great-models-fail-lessons-from-9-years-of-deploying-ml-models---megan-robertson-742qhwjzr#faq","mainEntity":[{"@type":"Question","name":"What is model drift and what are the different types of it?","acceptedAnswer":{"@type":"Answer","text":"Model drift is the phenomenon where a machine learning model's performance degrades over time because the world it was trained on changes. There are four types: concept drift (the relationship between input and target variables changes, as in evolving credit card fraud patterns), data drift (input feature distributions change, e.g. seasonal products), label drift (the target variable's distribution shifts, e.g. a sudden spike in churn), and reality drift (a sudden, drastic real-world shift with no prior signal in the data, like COVID-19). Track evolving strategies for detecting and handling model drift as production ML practices change on daily.dev."}},{"@type":"Question","name":"What are common reasons a well-performing machine learning model fails to make it into production?","acceptedAnswer":{"@type":"Answer","text":"Two dominant reasons are poor project scoping and the fact that the world changes over time. Poor scoping includes building without consulting stakeholders, evaluating success on vibes instead of quantitative KPIs, ignoring budget and compute costs, and delivering an overcomplicated solution when a simpler model performs nearly as well. Even a mathematically strong model can end up unused if its value doesn't clearly exceed its cost and risk. Compare real-world lessons on shipping ML projects successfully as new case studies surface on daily.dev."}},{"@type":"Question","name":"How can I detect and fix data drift or model performance degradation in a production ML pipeline?","acceptedAnswer":{"@type":"Answer","text":"Detection relies on monitoring summary statistics and statistical tests like the chi-square and Kolmogorov-Smirnov tests to compare current data distributions against historical baselines, alongside tracking model metrics such as MAE, MSE, precision, recall, F1, or semantic similarity for language models. Fixes include redoing feature engineering, retraining on more recent data, switching to a different algorithm, or retiring the model entirely if it no longer serves its purpose. Follow practical monitoring and retraining strategies for production ML systems on daily.dev."}}]} ```