--- title: "The 3 AM on-call problem: from \"find the bug\" to \"ship the fix\"" url: https://daily.dev/posts/the-3-am-on-call-problem-from-find-the-bug-to-ship-the-fix--gsbn9sfr1 source_url: https://daily.dev/posts/the-3-am-on-call-problem-from-find-the-bug-to-ship-the-fix--gsbn9sfr1 type: freeform source: "sourabh kumawat" author: "sourabh kumawat" published: 2026-06-21T12:40:09.607Z updated: 2026-06-21T12:40:57.230Z reading_time: 3 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # The 3 AM on-call problem: from "find the bug" to "ship the fix" **[sourabh kumawat](https://daily.dev/sources/2hn5ag8enhsaffiqxdkl8)** · [@sourabhkumawat](https://daily.dev/sourabhkumawat) · 3 min read · 0 upvotes · 0 comments ## Summary On-call incidents are costly not just due to downtime but because engineers waste time rebuilding context across fragmented tools at 3 AM. A new category of 'AI SRE' tooling aims to close the full loop from detection to fix. Tools like HealOps correlate signals across logs, metrics, traces, and configs, generate hypotheses in parallel, and produce a pull request with a diff, regression test, and evidence trail — rather than auto-deploying changes. This keeps humans in control of the merge button while removing the toil of manual investigation. Side effects include automatic runbook generation and regression checks that reduce repeat incidents. Key caveats: broad read access to sensitive systems requires careful scoping, hypothesis quality depends on observability data quality, and a competent human reviewer is still essential. ## Content If you've ever been on-call, you know the pattern. A page fires at 3 AM. You open your laptop half-awake and start the same ritual: pull traces from one tool, jump to another for metrics, dig into a third for the stack trace. The person who actually understands this corner of the system logged off hours ago, so you leave a Slack thread nobody will answer until morning. Eventually you reproduce something locally, guess at what changed, and ship a quick patch just so you can go back to bed. The real cost of an incident usually isn't the outage itself. It's the human attention spent rebuilding context from scratch, every single time, and the fact that a sleep-deprived "patch" rarely becomes a durable fix. That's the gap that a new wave of "AI SRE" tooling is trying to close. ![healops-oncall-tax.png](https://media.daily.dev/image/upload/s--cErvrot0--/f_auto/v1782042290/ugc/content_08d49add-dce7-4085-a3c2-e9069b031d78?_a=BAMAMiWQ0) So how does this actually get solved? The interesting shift is moving the agent's job from "find the root cause" to "close the loop." Most tooling stops at a diagnosis; the tired human still has to translate that into a diff. The newer approach treats investigation and remediation as the same task. A tool like HealOps ([healops.ai](http://healops.ai)) frames it as a loop with a few distinct stages: detect the alert, correlate signals across logs/metrics/traces/configs, form candidate hypotheses, run those hypotheses in parallel against your observability stack, then produce the fix. Here's how they break it down: ![healops-heal-loop.png](https://media.daily.dev/image/upload/s--r_pUw6kO--/f_auto/v1782042316/ugc/content_4efa49e5-1c77-4f7e-86c6-f31aeddae156?_a=BAMAMiWQ0) The part that stands out to me is the last mile. Instead of auto-deploying a change (which is how a lot of "auto-remediation" earns its bad reputation), the fix lands as a pull request on your repo — with the diff, a regression test that reproduces the incident, and the evidence trail behind the reasoning. Your reviewer still owns the merge button, so production stays behind human gates. The agent just removes the half-asleep typing. There's a nice side effect to closing the loop this way: every resolved incident can become a runbook entry and a regression check, so the same failure shape is less likely to page you twice. Over time the on-call noise gets quieter on purpose, instead of accumulating tribal knowledge in one senior engineer's head. A few honest caveats worth thinking about before adopting anything in this category: you're giving a tool read access to a lot of sensitive systems (logs, traces, source), so scope and permissions matter; "evidence-backed" hypotheses are only as good as your observability data; and a PR still needs a competent human reviewer — this lowers toil, it doesn't remove the need to understand your own system. Curious what folks here think: would you let an agent open PRs against your prod repos if a human still merges? Where's the line for you between "useful on-call leverage" and "too much autonomy"? ## Similar posts on daily.dev - [Beyond the Runbook: How to Scale SRE Operations for Cloud-Native Infrastructure](https://daily.dev/posts/beyond-the-runbook-how-to-scale-sre-operations-for-cloud-native-infrastructure-gmmvjkfsg) · Cloud Native Now · 0 upvotes · 0 comments - [3 steps to escaping the “break-fix” trap](https://daily.dev/posts/3-steps-to-escaping-the-break-fix-trap-2qkhuqmcb) · The New Stack · 0 upvotes · 0 comments --- [View this post on daily.dev](https://daily.dev/posts/the-3-am-on-call-problem-from-find-the-bug-to-ship-the-fix--gsbn9sfr1)