Sentry built a Claude Routine that connects through the Sentry MCP server to automatically triage roughly 800 overnight conversations from Seer, its AI agent for error triage and fixing. The routine pulls aggregate stats, samples errored tool spans and full conversations, checks whether the agent's verdict matches its reasoning, searches Linear for existing tickets, and files new ones for unresolved issues. It surfaced problems like malformed repo_name values causing repeated 'repository not found' retries and unexplained 30-56s hangs before internal errors. Sampling both successful and failed conversations proved important, since some conversations completed without tool errors yet still reached wrong conclusions. Setting it up just requires Agent Tracing, a Claude Routine prompt, and the Sentry MCP connector enabled.
Table of contents
The problemThe Claude RoutineWhat this looks like in practiceLessons learnedHow to do this yourselfQuestions this post answers
How can I automatically triage overnight AI agent conversation logs instead of reviewing them manually?
Set up a Claude Routine that connects through the Sentry MCP server to query Agent Tracing data each morning. It pulls aggregate conversation and tool-error stats, samples failing and successful conversation spans, checks whether the agent's verdict matches its reasoning, searches Linear for existing tickets, and files new ones for unresolved issues, all from a single plain-language prompt. daily.dev surfaces practical patterns like this for teams building agent observability into their workflows.
What kinds of hidden bugs can show up when sampling AI agent tool-call traces at scale?
Sampling roughly 400-550 conversations with around 11,000 tool calls revealed a 21% tool-error rate, mostly self-corrected retries. Root causes included malformed repo_name values (a monorepo subpath appended onto the real slug causing repeated 'repository not found' errors) and guessed-wrong project_slug values that self-corrected on retry, plus two unexplained 30-56 second hangs before an opaque internal error. engineers debugging flaky agent tool calls can track patterns like these through daily.dev.