--- title: We sized a guard from the incident that caused it. The... description: An engineer recounts a post-incident guard that failed to catch a smaller recurrence of the same vendor-data error: 24 bad rows worth about $3.0B slipped... canonical: https://daily.dev/posts/we-sized-a-guard-from-the-incident-that-caused-it-the-next-24-bad-rows-all-fit-under-both-threshold-rebriju2b twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: We sized a guard from the incident that caused it. The next 24 bad rows all fit under both thresholds. | daily.dev og:description: An engineer recounts a post-incident guard that failed to catch a smaller recurrence of the same vendor-data error: 24 bad rows worth about $3.0B slipped... og:url: https://daily.dev/posts/we-sized-a-guard-from-the-incident-that-caused-it-the-next-24-bad-rows-all-fit-under-both-threshold-rebriju2b og:image: https://api.daily.dev/og/posts/REBriJu2B.png og:image:alt: We sized a guard from the incident that caused it. The next 24 bad rows all fit under both thresholds. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # We sized a guard from the incident that caused it. The next 24 bad rows all fit under both thresholds. **[AlphAI](https://daily.dev/sources/alphai)** · [@mikhailmakeev](https://daily.dev/mikhailmakeev) · 4 min read · 0 upvotes · 0 comments ## Summary An engineer recounts a post-incident guard that failed to catch a smaller recurrence of the same vendor-data error: 24 bad rows worth about $3.0B slipped through because both sanity ceilings were tuned to the original incident's magnitude rather than the error class. A deeper problem emerged with the recovery formula used to fix suspected 'lot aggregate' errors: dividing price by shares was correct for one error shape but silently produced a plausible, wrong value for another (a decimal-shift error), producing a fabricated number nine times the real price. Three subsequent dry runs against production broke three different fix designs the unit test suite never caught. Lessons drawn: generalize guards to the error class, corroborate any recovery formula before trusting it, emit NULL instead of a plausible fabrication when nothing corroborates, use a detection mechanism independent of the guard's own predicate, and dry-run against production before trusting a design. ## Content Three weeks after we shipped a guard against bad vendor data, the same error came back at a smaller size and walked straight through it. Twenty-four rows, about $3.0B of value that never existed, and every one of them sat under both of the guard's thresholds. The scoring is what makes this expensive. Our relevance score for these events is deterministic and keys on the dollar value of the event, so an error doesn't just get in, it gets promoted. The wronger the number, the higher the row ranks. The five worst sat at the top of the feed at relevance 9. The guard we had was two independent ceilings: ```python _PRICE_SANITY_CEILING = Decimal(1_000_000) # per share _EVENT_VALUE_CEILING = Decimal(1_000_000_000) # per event over_per_share = price > _PRICE_SANITY_CEILING over_event = shares * price > _EVENT_VALUE_CEILING if not over_per_share and not over_event: return price ``` Both numbers came from the incident that prompted them: one filing at $1,230,435 per share and $18.46B for the event, which clears both by orders of magnitude. Here is what came next: ```text ticker shares filed price event value < $1M/share? < $1B/event? NMM 1,131 $748,119.00 $846.1M yes yes MFG 2,642 $127,239.37 $336.2M yes yes STNG 2,500 $191,325.00 $478.3M yes yes ``` The check never ran. I had written a guard that catches the incident, not the class. The second surprise was worse, and it is the part I would carry to any other pipeline. We had a recovery formula: if the price field looks like a lot aggregate, divide by shares. For one error shape that is exactly right. Thirteen rows in one batch all divided to the same $48.1602, and a fourteenth row in the same filing had been filed correctly at that very price, so the corroboration was sitting right there. For the other shape it is confidently wrong. One filing read: ```text date=2026-07-23 shares=1131 price=748119 # the broken row date=2026-07-24 shares=1084 price=79.019 date=2026-07-27 shares=1053 price=77.6356 ``` 748119 is $74.8119 with the decimal moved four places. The neighbouring days in the same filing say so, and the fake total is the true one times ten thousand. Run the recovery formula on it and you get $661.47, about nine times the real price. A missing number is honest. An invented one that looks plausible is a lie the rest of the system cannot detect. Then production had opinions. Three dry runs against the live database killed three designs in a row, and the 125 unit tests the fix already carried had caught none of them: 1. A single nominal row of $0.01 became the anchor for the whole ticker, which would have zeroed out a real $241.55 sale. 2. A five-row batch that was internally consistent anchored itself and never flagged, because every row in it divided to the same wrong price. 3. A seven-tranche error spread over two weeks defeated a date-based heuristic, which had been reasonable right up until the errors arrived on different days. What I would take from this into anything that ingests third-party data: - Name the error class, then ask what its small version looks like. A ceiling tuned to the case in front of you is a ceiling the next instance walks under. - Corroborate a recovery formula before you use it. `price / shares` was right and nine times wrong on the same day. - When nothing corroborates, write NULL. Downstream can handle a gap. It cannot handle a plausible fabrication. - Put the detector on a different mechanism than the guard. Sweeping for bad rows with the guard's own predicate confirms a clean database that isn't. - Dry run against production before you trust the design, and treat every discrepancy as a design input rather than a row to whitelist. One more that only shows up in the data: the same filer who caused the first incident filed the same shape again eighteen days later, and we published it again. Whatever you build, assume the upstream is not going to stop. ## Similar posts on daily.dev - [Count Is Not Risk: Ranking Data Issues by What They Actually Cost](https://daily.dev/posts/count-is-not-risk-ranking-data-issues-by-what-they-actually-cost-esovtfaud) · SQL Authority · 0 upvotes · 0 comments - [The Cheapest Insurance in Software](https://daily.dev/posts/the-cheapest-insurance-in-software-tk2aaiubp) · CodeGood · 2 upvotes · 0 comments - [Bug Detection Blind Spots in AI Coding Harnesses \(GStack and Beyond\)](https://daily.dev/posts/bug-detection-blind-spots-in-ai-coding-harnesses-gstack-and-beyond--5k8lonjtf) · Towards Data Science · 0 upvotes · 0 comments - [It Passed Every Test and Still Broke: Four Signals Validation Never Catches](https://daily.dev/posts/it-passed-every-test-and-still-broke-four-signals-validation-never-catches-pzcsecpac) · SQL Authority · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#python](https://daily.dev/tags/python), [#data-quality](https://daily.dev/tags/data-quality) [View this post on daily.dev](https://daily.dev/posts/we-sized-a-guard-from-the-incident-that-caused-it-the-next-24-bad-rows-all-fit-under-both-threshold-rebriju2b) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/we-sized-a-guard-from-the-incident-that-caused-it-the-next-24-bad-rows-all-fit-under-both-threshold-rebriju2b","headline":"We sized a guard from the incident that caused it. The next 24 bad rows all fit under both thresholds.","text":"An engineer recounts a post-incident guard that failed to catch a smaller recurrence of the same vendor-data error: 24 bad rows worth about $3.0B slipped through because both sanity ceilings were tuned to the original incident's magnitude rather than the error class. A deeper problem emerged with the recovery formula used to fix suspected 'lot aggregate' errors: dividing price by shares was correct for one error shape but silently produced a plausible, wrong value for another (a decimal-shift error), producing a fabricated number nine times the real price. Three subsequent dry runs against production broke three different fix designs the unit test suite never caught. Lessons drawn: generalize guards to the error class, corroborate any recovery formula before trusting it, emit NULL instead of a plausible fabrication when nothing corroborates, use a detection mechanism independent of the guard's own predicate, and dry-run against production before trusting a design.","url":"https://daily.dev/posts/we-sized-a-guard-from-the-incident-that-caused-it-the-next-24-bad-rows-all-fit-under-both-threshold-rebriju2b","datePublished":"2026-09-29T05:02:47.674Z","dateModified":"2026-09-29T05:03:11.259Z","author":{"@type":"Person","name":"Mikhail Makeev","url":"https://daily.dev/mikhailmakeev","image":"https://lh3.googleusercontent.com/a/ACg8ocIjcvDU1gmJ9-KJDeTYqY2rMdl5QxcWYtLTGSXrj8ubF4nwar_w=s96-c","description":"Owner of alphai.io","interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":3650}},"image":"https://media.daily.dev/image/upload/s--GFQrVAKD--/f_auto/v1790658167/posts/REBriJu2B?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/squads/alphai","name":"AlphAI"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AlphAI","item":"https://daily.dev/squads/alphai"},{"@type":"ListItem","position":3,"name":"We sized a guard from the incident that caused it. The next 24 bad rows all fit under both thresholds."}]} ```