--- title: From spot fixes to measured gains: improving document... description: A fix that halved errors on one field increased total extraction errors by 9% across the system. The authors built a workflow around ai_extract and MLflow... canonical: https://daily.dev/posts/from-spot-fixes-to-measured-gains-improving-document-extraction-accuracy-with-ai-extract-and-mlflow-vkmp90a1m twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: From spot fixes to measured gains: improving document extraction accuracy with ai_extract and MLflow Prompt Registry | daily.dev og:description: A fix that halved errors on one field increased total extraction errors by 9% across the system. The authors built a workflow around ai_extract and MLflow... og:url: https://daily.dev/posts/from-spot-fixes-to-measured-gains-improving-document-extraction-accuracy-with-ai-extract-and-mlflow-vkmp90a1m og:image: https://api.daily.dev/og/posts/VKMp90a1m.png og:image:alt: From spot fixes to measured gains: improving document extraction accuracy with ai_extract and MLflow Prompt Registry og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # From spot fixes to measured gains: improving document extraction accuracy with ai_extract and MLflow Prompt Registry **[databricks](https://daily.dev/sources/databricks)** · 13 min read · 1 upvotes · 0 comments ## Summary A fix that halved errors on one field increased total extraction errors by 9% across the system. The authors built a workflow around ai_extract and MLflow Prompt Registry to avoid this: version prompts and schemas separately from pipeline code, freeze a reviewer-approved test set, and run the production configuration 3-5 times to establish a noise floor before promoting changes. Restructuring the prompt and schema cut overall errors 16%, precision mode alone cut them 24%, and combining both cut errors 32% with revision-field errors down 64%. Genie Code can automate the evaluation and hill-climbing loop. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why did fixing one field's extraction accuracy make overall ai_extract errors worse? A schema change for the 'revision' field cut errors on that field roughly in half, but total extraction errors across all fields rose 9% because the model started inventing values in unrelated line-item fields like schedule details. Editing one field's description changes how the model reads the entire schema, not just that field, so single-field fixes need full-system evaluation before shipping. _Anyone iterating on LLM extraction prompts can track these regression trade-offs through daily.dev._ ### How do you measure whether a change to an LLM extraction prompt is actually an improvement and not just noise? Run the production configuration 3-5 times on a frozen, reviewer-approved test set and compute the mean accuracy and standard deviation (sigma) as a noise floor; require a candidate's improvement to exceed 2 sigma before counting it as meaningful. Also track coverage (share of non-null ground-truth fields with a prediction) alongside accuracy, since a candidate that leaves hard fields blank can look falsely more accurate. _Engineers validating LLM pipeline changes can follow this kind of evaluation rigor via daily.dev._ ### Should I put extraction rules in the ai_extract schema or in the global instructions prompt? Put rules that define a single field, like formatting a revision number or using null when absent, in the JSON schema description; reserve the global instructions prompt for rules about interpreting the whole document, such as preferring header fields over footers or listing forbidden sources. Keeping field rules in the schema makes the extraction contract easier to inspect instead of becoming a list of unrelated prompt exceptions. _Teams structuring ai_extract pipelines can keep this schema-versus-prompt split straight with daily.dev._ ## Similar posts on daily.dev - [How engineering teams are gaining market edge through systematic AI prompting](https://daily.dev/posts/how-engineering-teams-are-gaining-market-edge-through-systematic-ai-prompting-abix52usw) · SD Times · 1 upvotes · 0 comments - [How Automated Prompt Optimization Unlocks Quality Gains for ML Kit’s GenAI Prompt API](https://daily.dev/posts/how-automated-prompt-optimization-unlocks-quality-gains-for-ml-kit-s-genai-prompt-api-q8j6xn1gb) · Android Developers Blog · 0 upvotes · 0 comments - [Prompt Engineering Fails Quietly — Prompt Regression Is Why](https://daily.dev/posts/prompt-engineering-fails-quietly-prompt-regression-is-why-niy5iiaz2) · Towards Data Science · 2 upvotes · 1 comments - [Cortex AI Function Studio: Why “Vibes-Based” Prompt Engineering Has an Expiration Date](https://daily.dev/posts/cortex-ai-function-studio-why-vibes-based-prompt-engineering-has-an-expiration-date-oivnbbfsc) · Dev Genius · 1 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#llm](https://daily.dev/tags/llm), [#prompt-engineering](https://daily.dev/tags/prompt-engineering), [#databricks](https://daily.dev/tags/databricks) [View this post on daily.dev](https://daily.dev/posts/from-spot-fixes-to-measured-gains-improving-document-extraction-accuracy-with-ai-extract-and-mlflow-vkmp90a1m) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"From spot fixes to measured gains: improving document extraction accuracy with ai_extract and MLflow Prompt Registry","url":"https://daily.dev/posts/from-spot-fixes-to-measured-gains-improving-document-extraction-accuracy-with-ai-extract-and-mlflow-vkmp90a1m","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/from-spot-fixes-to-measured-gains-improving-document-extraction-accuracy-with-ai-extract-and-mlflow-vkmp90a1m"},"datePublished":"2026-10-27T17:40:15.275Z","dateModified":"2026-10-27T17:53:19.181Z","description":"A fix that halved errors on one field increased total extraction errors by 9% across the system. The authors built a workflow around ai_extract and MLflow...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8d3cd38c947fc1e81c785f71742a1e42?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8d3cd38c947fc1e81c785f71742a1e42?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"databricks","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"databricks","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/fa7aa720f2db4d1eba826814730482c8","url":"https://daily.dev/sources/databricks"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/from-spot-fixes-to-measured-gains-improving-document-extraction-accuracy-with-ai-extract-and-mlflow-vkmp90a1m","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,llm,prompt-engineering,databricks","timeRequired":"PT13M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"databricks","item":"https://daily.dev/sources/databricks"},{"@type":"ListItem","position":3,"name":"From spot fixes to measured gains: improving document extraction accuracy with ai_extract and MLflow Prompt Registry"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/from-spot-fixes-to-measured-gains-improving-document-extraction-accuracy-with-ai-extract-and-mlflow-vkmp90a1m#faq","mainEntity":[{"@type":"Question","name":"Why did fixing one field's extraction accuracy make overall ai_extract errors worse?","acceptedAnswer":{"@type":"Answer","text":"A schema change for the 'revision' field cut errors on that field roughly in half, but total extraction errors across all fields rose 9% because the model started inventing values in unrelated line-item fields like schedule details. Editing one field's description changes how the model reads the entire schema, not just that field, so single-field fixes need full-system evaluation before shipping. Anyone iterating on LLM extraction prompts can track these regression trade-offs through daily.dev."}},{"@type":"Question","name":"How do you measure whether a change to an LLM extraction prompt is actually an improvement and not just noise?","acceptedAnswer":{"@type":"Answer","text":"Run the production configuration 3-5 times on a frozen, reviewer-approved test set and compute the mean accuracy and standard deviation (sigma) as a noise floor; require a candidate's improvement to exceed 2 sigma before counting it as meaningful. Also track coverage (share of non-null ground-truth fields with a prediction) alongside accuracy, since a candidate that leaves hard fields blank can look falsely more accurate. Engineers validating LLM pipeline changes can follow this kind of evaluation rigor via daily.dev."}},{"@type":"Question","name":"Should I put extraction rules in the ai_extract schema or in the global instructions prompt?","acceptedAnswer":{"@type":"Answer","text":"Put rules that define a single field, like formatting a revision number or using null when absent, in the JSON schema description; reserve the global instructions prompt for rules about interpreting the whole document, such as preferring header fields over footers or listing forbidden sources. Keeping field rules in the schema makes the extraction contract easier to inspect instead of becoming a list of unrelated prompt exceptions. Teams structuring ai_extract pipelines can keep this schema-versus-prompt split straight with daily.dev."}}]} ```