Monitoring LLM agents in production requires fundamentally different approaches than traditional APM tools. Agents accept unbounded natural language inputs, exhibit non-deterministic behavior, and operate through multi-step reasoning chains — making system metrics insufficient on their own. Effective agent observability requires capturing full prompt-response pairs, multi-turn context, and agent trajectories. Two complementary evaluation strategies are discussed: annotation queues for structured human review of high-value traces, and LLM-as-a-judge evaluators that can assess thousands of traces automatically (with recommended 10-20% sampling to manage cost). LangSmith's Insights Agent provides automated clustering to surface usage patterns and failure modes. The post also explains why general-purpose tools like Datadog fall short in three areas: handling natural language payloads at scale, integrating with the development feedback loop, and supporting cross-functional teams beyond SRE/DevOps.
