--- title: I built a monitor. Nearly 200 engineers turned it into a... description: An engineering lead recounts four months of building and running an internal AI on-call agent that grew from a personal tool into infrastructure adopted by 36... canonical: https://daily.dev/posts/i-built-a-monitor-nearly-200-engineers-turned-it-into-a-colleague--j0ncs7fqu twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: I built a monitor. Nearly 200 engineers turned it into a colleague. | daily.dev og:description: An engineering lead recounts four months of building and running an internal AI on-call agent that grew from a personal tool into infrastructure adopted by 36... og:url: https://daily.dev/posts/i-built-a-monitor-nearly-200-engineers-turned-it-into-a-colleague--j0ncs7fqu og:image: https://api.daily.dev/og/posts/j0NCs7FQU.png og:image:alt: I built a monitor. Nearly 200 engineers turned it into a colleague. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # I built a monitor. Nearly 200 engineers turned it into a colleague. **[monday Engineering](https://daily.dev/sources/monday)** · 15 min read · 1 upvotes · 0 comments ## Summary An engineering lead recounts four months of building and running an internal AI on-call agent that grew from a personal tool into infrastructure adopted by 36 agents across 28 teams. Key lessons include picking a bounded repeatable problem, giving the agent a memorable name to drive adoption, dropping proactive monitoring in favor of a reactive help-desk model since usage data showed nobody wanted proactive alerts, treating the prompt as a strict runbook with numeric SLAs, explicit field IDs, a closed failure taxonomy, and asymmetric read/write permissions based on blast radius, and offloading rarely-needed procedural knowledge into loadable skills rather than bloating the prompt. It also covers failure modes: the agent cannot diagnose its own outages because silence and failure look identical, so aliveness checks must live outside the agent. Session volume grew from 104 to 243 per week with a 3.4% failure rate across 1,133 sessions. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Should an on-call AI agent be proactive or reactive by default? Reactive first. A proactive design running hourly anomaly sweeps and three scheduled jobs was shut off after eleven days because nobody wanted a watchtower; what people actually needed was a help desk that never goes home, available the moment someone hit a problem rather than surfacing issues nobody had reported. Ship the reactive path and let real usage tell you whether proactive automation is worth adding. _Teams weighing agent design trade-offs like this compare real-world write-ups on daily.dev before committing to an architecture._ ### How do you stop an AI on-call agent from hallucinating a root cause commit? Force classification before diagnosis and make an honest null result count as success. The agent must sort every incident into one of four boxes (code regression, prompt/skill gap, external dependency, working as intended) before naming a cause, and it may attribute a commit only in the code-regression class. A retention-honesty rule also blocks it from blaming the oldest commit in a query window when that commit only appears earliest because of a data retention boundary. _Anyone hardening an incident-response agent can track prompt-design patterns like this on daily.dev._ ### How do you know if a monitoring AI agent is silently failing versus correctly reporting no issues? You can't tell from inside the agent, because a design instructed to stay silent unless something's wrong makes silence and failure produce the same signal. A database connection pool exhaustion once took the agent down and its absence was indistinguishable from an all-clear. The fix is external: something outside the agent itself must independently verify it is alive, since prompt engineering cannot solve this. _Engineers designing agent uptime checks can follow related architecture write-ups on daily.dev._ ## Similar posts on daily.dev - [Three Years of Building Agents in Production \(Part 2\)](https://daily.dev/posts/three-years-of-building-agents-in-production-part-2--9uvhtnda1) · Mabl Engineering Blog · 2 upvotes · 1 comments - [From firefighting to building: How AI agents restored our team’s core productivity](https://daily.dev/posts/from-firefighting-to-building-how-ai-agents-restored-our-team-s-core-productivity-zfyvdslnl) · Grab Tech Blog · 2 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents), [#observability](https://daily.dev/tags/observability), [#prompt-engineering](https://daily.dev/tags/prompt-engineering) [View this post on daily.dev](https://daily.dev/posts/i-built-a-monitor-nearly-200-engineers-turned-it-into-a-colleague--j0ncs7fqu) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"I built a monitor. Nearly 200 engineers turned it into a colleague.","url":"https://daily.dev/posts/i-built-a-monitor-nearly-200-engineers-turned-it-into-a-colleague--j0ncs7fqu","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/i-built-a-monitor-nearly-200-engineers-turned-it-into-a-colleague--j0ncs7fqu"},"datePublished":"2026-09-01T13:39:46.291Z","dateModified":"2026-09-01T13:40:14.667Z","description":"An engineering lead recounts four months of building and running an internal AI on-call agent that grew from a personal tool into infrastructure adopted by 36...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5a5d178d450d464901d1a158633098cb?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5a5d178d450d464901d1a158633098cb?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"monday Engineering","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"monday Engineering","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/3bce2c4601a8468097dd077a4a9e1417","url":"https://daily.dev/sources/monday"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/i-built-a-monitor-nearly-200-engineers-turned-it-into-a-colleague--j0ncs7fqu","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,ai-agents,observability,prompt-engineering","timeRequired":"PT15M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"monday Engineering","item":"https://daily.dev/sources/monday"},{"@type":"ListItem","position":3,"name":"I built a monitor. Nearly 200 engineers turned it into a colleague."}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/i-built-a-monitor-nearly-200-engineers-turned-it-into-a-colleague--j0ncs7fqu#faq","mainEntity":[{"@type":"Question","name":"Should an on-call AI agent be proactive or reactive by default?","acceptedAnswer":{"@type":"Answer","text":"Reactive first. A proactive design running hourly anomaly sweeps and three scheduled jobs was shut off after eleven days because nobody wanted a watchtower; what people actually needed was a help desk that never goes home, available the moment someone hit a problem rather than surfacing issues nobody had reported. Ship the reactive path and let real usage tell you whether proactive automation is worth adding. Teams weighing agent design trade-offs like this compare real-world write-ups on daily.dev before committing to an architecture."}},{"@type":"Question","name":"How do you stop an AI on-call agent from hallucinating a root cause commit?","acceptedAnswer":{"@type":"Answer","text":"Force classification before diagnosis and make an honest null result count as success. The agent must sort every incident into one of four boxes (code regression, prompt/skill gap, external dependency, working as intended) before naming a cause, and it may attribute a commit only in the code-regression class. A retention-honesty rule also blocks it from blaming the oldest commit in a query window when that commit only appears earliest because of a data retention boundary. Anyone hardening an incident-response agent can track prompt-design patterns like this on daily.dev."}},{"@type":"Question","name":"How do you know if a monitoring AI agent is silently failing versus correctly reporting no issues?","acceptedAnswer":{"@type":"Answer","text":"You can't tell from inside the agent, because a design instructed to stay silent unless something's wrong makes silence and failure produce the same signal. A database connection pool exhaustion once took the agent down and its absence was indistinguishable from an all-clear. The fix is external: something outside the agent itself must independently verify it is alive, since prompt engineering cannot solve this. Engineers designing agent uptime checks can follow related architecture write-ups on daily.dev."}}]} ```