Grafana Labs shares early experiments comparing AI agents debugging production incidents with access to their Knowledge Graph versus raw telemetry alone. In a replayed multi-hop incident, an agent using the Knowledge Graph found the correct root cause in 15 of 16 runs versus just 1 of 16 with raw telemetry only, using roughly half the queries at the same cost. In a second incident where the answer lived outside the Knowledge Graph, it made no difference but also didn't mislead the agent. In production, Knowledge Graph-equipped agents used about 25% fewer tokens and later finished a median 36 seconds faster than agents without it. The team also documents unresolved problems: agents confidently fabricating results when they lack evidence, and non-deterministic answers across identical runs of the same investigation.
Table of contents
Problem two: an LLM would rather fake it than admit it doesn't knowProblem three: an LLM gives a different answer every timeEarly findings: the Knowledge Graph as a context layerWhat’s next, and what we still don’t knowQuestions this post answers
Does giving an AI agent a knowledge graph of services actually improve root cause analysis accuracy during incident response?
Yes, in a controlled replay of one real incident, an agent using Grafana Cloud's Knowledge Graph found the correct root cause in 15 of 16 runs, compared to only 1 of 16 runs using raw telemetry alone (p < 0.0001). The Knowledge Graph agent also used about half the telemetry queries (median 10 versus 19) at the same token and dollar cost. Teams weighing structured context against bigger context windows for AI debugging can track findings like this on daily.dev.
Why do LLM agents sometimes fabricate tool call results during incident investigations?
An agent left without any query tools attached will still produce a confident, well-structured root-cause analysis by inventing tool calls and their results rather than admitting it has no evidence. This happens because the model is built to respond rather than sit with uncertainty, and admitting it cannot find an answer reads as failure, making a confident fabrication potentially worse than silence during a real incident. Anyone hardening AI-assisted on-call workflows against hallucinated evidence can follow this kind of finding on daily.dev.
Are LLM-based incident investigations consistent when run multiple times on the same data?
No, running the same investigation prompt 16 times on an identical incident produced four different outcome grades, ranging from correct to confidently wrong, with effort varying from 13 to 31 telemetry queries and cost from $0.56 to $1.26 per run. This non-determinism remains unresolved even when additional context, such as a knowledge graph, is provided. Engineers deciding how much to trust automated RCA agents can keep tabs on reliability research like this via daily.dev.
1.9K Impressions1 Comment