--- title: Your JSON Is Valid but Your Data Is Wrong: Five Failure... description: Constrained decoding guarantees schema-valid JSON from LLMs, but valid structure doesn't mean correct data. Five failure modes slip through schema validation:... canonical: https://daily.dev/posts/your-json-is-valid-but-your-data-is-wrong-five-failure-modes-llm-structured-outputs-won-t-catch-ilmoiq1ob twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Your JSON Is Valid but Your Data Is Wrong: Five Failure Modes LLM Structured Outputs Won't Catch | daily.dev og:description: Constrained decoding guarantees schema-valid JSON from LLMs, but valid structure doesn't mean correct data. Five failure modes slip through schema validation:... og:url: https://daily.dev/posts/your-json-is-valid-but-your-data-is-wrong-five-failure-modes-llm-structured-outputs-won-t-catch-ilmoiq1ob og:image: https://api.daily.dev/og/posts/ilMOiQ1Ob.png og:image:alt: Your JSON Is Valid but Your Data Is Wrong: Five Failure Modes LLM Structured Outputs Won't Catch og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Your JSON Is Valid but Your Data Is Wrong: Five Failure Modes LLM Structured Outputs Won't Catch **[Towards Data Science](https://daily.dev/sources/tds)** · 10 min read · 1 upvotes · 1 comments ## Summary Constrained decoding guarantees schema-valid JSON from LLMs, but valid structure doesn't mean correct data. Five failure modes slip through schema validation: enum hallucination (semantically wrong but valid enum values), confident fabrication (plausible invented data in free-text fields, like a fake expense report generated from an elephant photo), cross-field contradiction (individually valid fields that conflict, like end dates before start dates), distributional collapse (confidence scores flatlining at safe defaults like 0.98), and array hallucination (fabricated entries instead of empty arrays). BAML benchmarks found unconstrained generation with post-hoc parsing hit 93.63% accuracy versus 91.37% for constrained decoding on the same model. A proposed three-layer defense combines schema validation, semantic validators (cross-field business logic, entropy monitoring), and uncertainty surfacing (confidence fields, LLM-as-judge verification) to catch what schema compliance alone misses. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does my LLM pipeline return schema-valid JSON that still contains wrong data? Schema validation only checks that fields are typed correctly, not that their values are semantically correct. Five failure modes commonly slip through: enum hallucination (valid but contextually wrong enum picks), confident fabrication (plausible invented values in free-text fields), cross-field contradiction (fields valid alone but inconsistent together, like a positive sentiment label paired with a 0.1 score), distributional collapse (confidence scores flatlining at safe defaults like 0.98), and array hallucination (fabricated entries instead of empty arrays). _daily.dev surfaces practical write-ups for teams hardening LLM extraction pipelines against silent failures._ ### Is constrained decoding more or less accurate than letting an LLM generate JSON freely and parsing it afterward? Unconstrained generation with post-hoc parsing was measured at 93.63% accuracy on function-calling tasks, compared to 91.37% for constrained decoding on the same model, according to BAML's benchmarks. Separate research found forcing structured output formats produced a 3-to-9 percentage point accuracy drop overall, and over 15 points on math reasoning tasks, because formatting pressure competes with reasoning quality. _Developers choosing between constrained decoding and resampling can track these trade-offs on daily.dev._ ### How can I detect if my LLM extraction pipeline is silently degrading even though outputs still validate against my schema? Watch three statistical signals rather than individual outputs: dropping entropy in a field that should vary (values clustering on one or two options), an empty-array rate near zero for arrays that should sometimes be empty, and downstream business metrics drifting even though the model version, prompt, and schema haven't changed. Any of these can indicate distributional collapse or fabrication that passes schema checks. _Teams monitoring LLM pipeline reliability can follow emerging detection techniques on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@kartiknvj** · 1 upvotes > Schema-valid but semantically wrong is the failure I trust least, because the green checkmark from constrained decoding hides it. The distributional-collapse point (confidence flatlining at 0.98) matches what I see, so I lean on cross-field validators rather than the model's own confidence score. Did the BAML accuracy edge hold on your harder schemas? ## Similar posts on daily.dev - [Your LLM Can Return Perfect JSON and Still Be Wrong](https://daily.dev/posts/your-llm-can-return-perfect-json-and-still-be-wrong-o3ogkexgv) · Towards Data Science · 1 upvotes · 0 comments - [Structured Outputs Create False Confidence](https://daily.dev/posts/structured-outputs-create-false-confidence-nukgaru4e) · Hacker News · 3 upvotes · 0 comments - [How to Get Reliable Structured Data Out of an LLM](https://daily.dev/posts/how-to-get-reliable-structured-data-out-of-an-llm-lzwbvavpz) · freeCodeCamp · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#pydantic](https://daily.dev/tags/pydantic) [View this post on daily.dev](https://daily.dev/posts/your-json-is-valid-but-your-data-is-wrong-five-failure-modes-llm-structured-outputs-won-t-catch-ilmoiq1ob) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Your JSON Is Valid but Your Data Is Wrong: Five Failure Modes LLM Structured Outputs Won't Catch","url":"https://daily.dev/posts/your-json-is-valid-but-your-data-is-wrong-five-failure-modes-llm-structured-outputs-won-t-catch-ilmoiq1ob","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/your-json-is-valid-but-your-data-is-wrong-five-failure-modes-llm-structured-outputs-won-t-catch-ilmoiq1ob"},"datePublished":"2026-09-01T15:48:10.379Z","dateModified":"2026-09-01T15:48:37.779Z","description":"Constrained decoding guarantees schema-valid JSON from LLMs, but valid structure doesn't mean correct data. Five failure modes slip through schema validation:...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0634cab65c9752da6258c6a64d996a7c?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0634cab65c9752da6258c6a64d996a7c?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Towards Data Science","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Towards Data Science","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/tds","url":"https://daily.dev/sources/tds"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/your-json-is-valid-but-your-data-is-wrong-five-failure-modes-llm-structured-outputs-won-t-catch-ilmoiq1ob","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"llm,pydantic","timeRequired":"PT10M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Towards Data Science","item":"https://daily.dev/sources/tds"},{"@type":"ListItem","position":3,"name":"Your JSON Is Valid but Your Data Is Wrong: Five Failure Modes LLM Structured Outputs Won't Catch"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/your-json-is-valid-but-your-data-is-wrong-five-failure-modes-llm-structured-outputs-won-t-catch-ilmoiq1ob","comment":[{"@type":"Comment","text":"Schema-valid but semantically wrong is the failure I trust least, because the green checkmark from constrained decoding hides it. The distributional-collapse point (confidence flatlining at 0.98) matches what I see, so I lean on cross-field validators rather than the model’s own confidence score. Did the BAML accuracy edge hold on your harder schemas?","datePublished":"2026-09-01T18:18:28.599Z","url":"https://daily.dev/posts/ilMOiQ1Ob#c-LaEEGeVxT","author":{"@type":"Person","name":"kartik-nvjk","url":"https://daily.dev/kartiknvj","image":"https://media.daily.dev/image/upload/s--3gGgsVCw--/f_auto/v1781456774/avatars/avatar_TvTVeiMdkRCqWUDullFmy?_a=BAMAMiWQ0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/your-json-is-valid-but-your-data-is-wrong-five-failure-modes-llm-structured-outputs-won-t-catch-ilmoiq1ob#faq","mainEntity":[{"@type":"Question","name":"Why does my LLM pipeline return schema-valid JSON that still contains wrong data?","acceptedAnswer":{"@type":"Answer","text":"Schema validation only checks that fields are typed correctly, not that their values are semantically correct. Five failure modes commonly slip through: enum hallucination (valid but contextually wrong enum picks), confident fabrication (plausible invented values in free-text fields), cross-field contradiction (fields valid alone but inconsistent together, like a positive sentiment label paired with a 0.1 score), distributional collapse (confidence scores flatlining at safe defaults like 0.98), and array hallucination (fabricated entries instead of empty arrays). daily.dev surfaces practical write-ups for teams hardening LLM extraction pipelines against silent failures."}},{"@type":"Question","name":"Is constrained decoding more or less accurate than letting an LLM generate JSON freely and parsing it afterward?","acceptedAnswer":{"@type":"Answer","text":"Unconstrained generation with post-hoc parsing was measured at 93.63% accuracy on function-calling tasks, compared to 91.37% for constrained decoding on the same model, according to BAML's benchmarks. Separate research found forcing structured output formats produced a 3-to-9 percentage point accuracy drop overall, and over 15 points on math reasoning tasks, because formatting pressure competes with reasoning quality. Developers choosing between constrained decoding and resampling can track these trade-offs on daily.dev."}},{"@type":"Question","name":"How can I detect if my LLM extraction pipeline is silently degrading even though outputs still validate against my schema?","acceptedAnswer":{"@type":"Answer","text":"Watch three statistical signals rather than individual outputs: dropping entropy in a field that should vary (values clustering on one or two options), an empty-array rate near zero for arrays that should sometimes be empty, and downstream business metrics drifting even though the model version, prompt, and schema haven't changed. Any of these can indicate distributional collapse or fabrication that passes schema checks. Teams monitoring LLM pipeline reliability can follow emerging detection techniques on daily.dev."}}]} ```