--- title: "Your incident response wasn’t built for AI" url: https://daily.dev/posts/your-incident-response-wasn-t-built-for-ai-a5d9qgdoc source_url: https://leaddev.com/ai/your-incident-response-wasnt-built-for-ai type: article source: "LeadDev" published: 2026-08-19T08:43:03.319Z updated: 2026-08-19T08:43:30.897Z tags: ["llm", "ai-agents", "observability", "sre"] reading_time: 7 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Your incident response wasn’t built for AI **[LeadDev](https://daily.dev/sources/leaddev)** · 7 min read · 0 upvotes · 0 comments ## Summary AI systems in production break the assumptions incident response relies on: failures aren't reproducible due to GPU batching nondeterminism, they return successful status codes while being wrong, and no team clearly owns the quality floor since product picks the model while platform holds the pager. The proposed fix splits SLOs into three tiers: service (availability/latency, unchanged), behavioral (task success, capped by the model's own training and reported per segment), and containment (the only tier a team can actually control - catch rate and time to disable). Practical recommendations include treating a kill switch at capability granularity as a launch gate, capturing full trace context instead of trying to reproduce failures, and rewriting postmortems to grade response quality rather than demand root-cause fixes that don't exist. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### why does my LLM give different outputs for the same input and temperature set to zero even in production GPU batching causes it. Inference servers batch concurrent requests onto the same GPU, and which requests land together depends on traffic you don't control, changing the order floating-point additions happen in. Since floating point addition isn't perfectly associative, scores shift in the fourth decimal place, which can flip which token wins when top candidates are close, cascading into a different output. _Teams debugging nondeterministic model behavior can find deeper technical breakdowns like this on daily.dev._ ### how should I structure SLOs for a production AI feature Split the SLO into three tiers: service (availability and latency, measured as before), behavioral (task success against a versioned golden set, sampled continuously and reported per segment rather than in aggregate), and containment (fraction of bad outputs caught before users see them, and time from detection to disablement). Only the containment tier belongs entirely to the team running it, since behavioral quality is capped by a training run they didn't commission. _Engineering leaders designing AI reliability practices track frameworks like this tiered SLO model on daily.dev._ ### why doesn't my monitoring catch AI model quality regressions Because a model regression often lands as a small percentage of requests concentrated in one customer segment, moving the aggregate quality metric by less than its own noise floor, so no dashboard displays it and the failure returns a 200 OK instead of an error. This means the escalation typically surfaces through account teams rather than paging, since availability tooling stays green through a correctness incident. _Teams building AI observability practices follow real incident patterns like this on daily.dev._ ## Similar posts on daily.dev - [So Agentic Systems Are Messing Up Your SLO Framework](https://daily.dev/posts/so-agentic-systems-are-messing-up-your-slo-framework-ueygqji6y) · DevOps.com · 1 upvotes · 0 comments - [What to do About AI’s Forced Rethink of Reliability in Modern DevOps](https://daily.dev/posts/what-to-do-about-ai-s-forced-rethink-of-reliability-in-modern-devops-aet0vsbfn) · DevOps.com · 0 upvotes · 0 comments - [AI Agent Reliability Engineering 2026: SLOs and Failure Modes](https://daily.dev/posts/ai-agent-reliability-engineering-2026-slos-and-failure-modes-gjjxsmprz) · Alex CloudStar · 1 upvotes · 0 comments - [A Successful Response And A Correct Response Are Not The Same Thing In Distributed Systems](https://daily.dev/posts/a-successful-response-and-a-correct-response-are-not-the-same-thing-in-distributed-systems-inp6tnno4) · Medium · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents), [#observability](https://daily.dev/tags/observability), [#sre](https://daily.dev/tags/sre) [View this post on daily.dev](https://daily.dev/posts/your-incident-response-wasn-t-built-for-ai-a5d9qgdoc)