Kelet AI built a production system that continuously diagnoses quality failures in AI agents by analyzing patterns across thousands of sessions — not individual traces. The core insight is that AI failures are fuzzy clusters that only emerge across many sessions, requiring multi-stage pipeline infrastructure rather than a simple agent loop. Built on Temporal, the system uses a four-level Workflow hierarchy: Session Workflows with debounce windows, parallel Signal Workflows for enrichment, an Agent Aggregation Workflow for cross-session hypothesis collection, and an Investigate Issue Workflow for root-cause clustering and prompt patch generation. Key Temporal primitives used include long-running state, durable execution, event-driven Signals, wait_condition, and continue_as_new. The system monitors itself in production via a Temporal plugin that propagates session context across Workers and Activities using OTel headers, with an infinite-loop guard to prevent self-monitoring from re-ingesting its own diagnostic sessions.
