AI agents require fundamentally different observability and evaluation approaches than traditional software because their behavior is non-deterministic and only emerges at runtime. Three core observability primitives are introduced: runs (single LLM call captures), traces (complete agent execution trajectories), and threads (multi-turn conversation sessions). Evaluation maps directly to these primitives at three granularities: single-step (unit tests for individual decisions), full-turn (end-to-end trajectory assessment), and multi-turn (context persistence across conversations). Production traces serve triple duty as debugging tools, offline evaluation datasets, and continuous online evaluation inputs. The key insight is that tracing and evaluation are inseparable for agents — production is not just where you catch bugs, but where you discover what to test for.
