Automating root cause analysis at scale: Multi-signal correlation for cloud native incident response
Atlassian engineers describe an automated root cause analysis (RCA) system that treats incident diagnosis as a multi-signal correlation problem across signal type, time, and service topology. The pipeline scopes the search using an OpenTelemetry-derived service dependency graph, detects anomalies independently in metrics, traces, and logs, correlates them temporally via sliding windows and sequence fingerprinting to deduplicate replayed failure patterns, then performs graph-based causal analysis to identify the likely fault origin and propagation path. Ranked hypotheses include confidence scores and human-readable narratives to build responder trust. The system feeds into a broader incident response platform with faulty-service paging, an AI copilot, and a feedback loop, with future work exploring iterative LLM-based orchestration and additional signal types like deployment events and feature flags.
Table of contents
The problem: Humans shouldn’t be correlation enginesOur approach: Treat RCA as a multi-signal correlation problemArchitecture: From raw telemetry to ranked hypothesesFitting into a broader reliability platformLessons learned and design trade-offsWhat’s nextKey takeawaysQuestions this post answers
How can I automatically correlate metrics, logs, and traces to find the root cause of a production incident?
Root cause analysis can be automated by treating it as a correlation problem across three dimensions: signal type, time, and service topology. Anomalies are detected independently per signal (metrics via MAD and percentile bands, traces via structural analysis, logs via embedding-based clustering), normalized into a common event schema, grouped into temporal correlation bundles using a sliding window, then traced through a service dependency graph via BFS to identify causal direction and rank hypotheses by combined temporal and path scores. See how daily.dev surfaces engineering deep dives like this on observability and incident response tooling.
How do you avoid generating duplicate incident hypotheses when the same failure keeps repeating during an outage?
Duplicate hypotheses can be eliminated using sequence fingerprinting, which computes a fingerprint from the ordered list of services in each anomaly path and collapses repeated occurrences of the same causal chain into a single correlation bundle with a replay count. This lets a system report that a failure pattern repeated dozens of times in a few minutes instead of generating a separate hypothesis for each occurrence, keeping the output readable for on-call responders. Engineers tuning noisy alerting pipelines can track more posts like this on daily.dev.
Why does OpenTelemetry matter for building an automated root cause analysis system?
OpenTelemetry provides the service dependency graph needed for causal inference by building span-level parent-child relationships from actual production traffic, giving a real-time picture of how services communicate rather than relying on outdated documentation. This graph is used to scope the blast radius during an incident and to traverse upstream for graph-based impact analysis, making consistent, correlated telemetry a prerequisite for multi-signal RCA. Track how teams instrument OpenTelemetry for real production use cases on daily.dev.