Databricks describes AI SRE, an internal AI-powered debugging agent that kicks off automatically when an incident fires, running parallel investigation tracks (platform health checks, service-level analysis, and runbook execution) to assemble diagnostic context before an on-call engineer even opens the alert. The system is built as a layered platform (primitives, API layer, core engine, application layer) so teams can encode their own runbooks as 'agentic runbooks' rather than relying on a single centralized agent. Design principles emphasize deterministic checks before open-ended LLM reasoning, transparency via traceable evidence, and graceful degradation when confidence is low. The tool now serves 150+ teams, 250+ weekly active users, and over 2,000 investigations daily, reportedly saving hours of debugging time and reducing MTTR.
Table of contents
Before AI SRE: The 2 AM ExperienceStarting with the Customer, Not the TechnologyIntroducing AI SREBuilding for Reliability in a Non-Deterministic WorldImpactWhat We LearnedWhat's NextJoin UsQuestions this post answers
What is Databricks' AI SRE system and how does it help with incident investigation?
AI SRE is an internal AI-powered debugging agent at Databricks that automatically begins investigating as soon as an incident fires, running three parallel tracks: platform health checks, service-level log/metric/trace analysis, and execution of team-specific agentic runbooks. It assembles a diagnostic summary before the on-call engineer even opens the alert, and supports over 150 teams with 250+ weekly active users running 2,000+ investigations daily. daily.dev surfaces engineering deep dives like this for teams designing AI-assisted incident response workflows.
How is Databricks' AI SRE platform architected into layers?
It uses four layers: primitives (raw operational data like metrics, logs, alerts, deployments), an API layer providing uniform authenticated access via purpose-specific APIs like Observability, Deployment, and Alerts APIs, a core engine handling orchestration and LLM-powered synthesis via a bot framework, and an application layer where the triage bot and third-party AI tools run. This separation lets data access and orchestration evolve independently of debugging workflows. engineers designing layered observability platforms can follow architecture writeups like this on daily.dev.
Why does Databricks run deterministic health checks before letting an LLM reason about an incident?
Because trust is critical during incident response, Databricks runs deterministic platform health checks and runbook steps first, and only lets the LLM synthesize and explain results rather than gather data on its own judgment. Every conclusion links back to underlying evidence like specific metrics, log lines, and deploy diffs so engineers can audit rather than blindly trust recommendations, and the system explicitly reports when it cannot determine a root cause with confidence. teams weighing how much autonomy to give AI agents in production can track these design tradeoffs on daily.dev.
12.6K Impressions2 Comments