Your incident response wasn’t built for AI
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
AI systems in production break the assumptions incident response relies on: failures aren't reproducible due to GPU batching nondeterminism, they return successful status codes while being wrong, and no team clearly owns the quality floor since product picks the model while platform holds the pager. The proposed fix splits SLOs into three tiers: service (availability/latency, unchanged), behavioral (task success, capped by the model's own training and reported per segment), and containment (the only tier a team can actually control - catch rate and time to disable). Practical recommendations include treating a kill switch at capability granularity as a launch gate, capturing full trace context instead of trying to reproduce failures, and rewriting postmortems to grade response quality rather than demand root-cause fixes that don't exist.
Table of contents
Your incident loop just lost step twoMore like thisThe failure returns a 200The fault line nobody drew on the org chartWhat to do insteadThe actual jobQuestions this post answers
why does my LLM give different outputs for the same input and temperature set to zero even in production
GPU batching causes it. Inference servers batch concurrent requests onto the same GPU, and which requests land together depends on traffic you don't control, changing the order floating-point additions happen in. Since floating point addition isn't perfectly associative, scores shift in the fourth decimal place, which can flip which token wins when top candidates are close, cascading into a different output. Teams debugging nondeterministic model behavior can find deeper technical breakdowns like this on daily.dev.
how should I structure SLOs for a production AI feature
Split the SLO into three tiers: service (availability and latency, measured as before), behavioral (task success against a versioned golden set, sampled continuously and reported per segment rather than in aggregate), and containment (fraction of bad outputs caught before users see them, and time from detection to disablement). Only the containment tier belongs entirely to the team running it, since behavioral quality is capped by a training run they didn't commission. Engineering leaders designing AI reliability practices track frameworks like this tiered SLO model on daily.dev.
why doesn't my monitoring catch AI model quality regressions
Because a model regression often lands as a small percentage of requests concentrated in one customer segment, moving the aggregate quality metric by less than its own noise floor, so no dashboard displays it and the failure returns a 200 OK instead of an error. This means the escalation typically surfaces through account teams rather than paging, since availability tooling stays green through a correctness incident. Teams building AI observability practices follow real incident patterns like this on daily.dev.