Running AI agents in production exposes gaps in traditional APM — agents don't crash with stack traces, they loop, hallucinate, and burn tokens silently. The post outlines a three-pillar observability approach: agent traces (using Langfuse for full decision history with nested spans and non-blocking delivery), cost monitoring (per-session and per-agent burn rates with proactive circuit breakers like iteration caps and loop detection), and immutable audit logs with PII redaction. Additional recommendations include a doctor-style diagnostic command for dependency health checks, automated trace analysis to flag anomalous sessions, and careful Prometheus label cardinality management. Key lessons: cost spikes are the best early-warning signal, traces are for debugging while metrics are for alerting, and automated anomaly flagging is essential at scale.
