Building an Evaluation Harness for Production AI Agents: A 12-Metric Framework From 100+ Deployments

A comprehensive 12-metric evaluation framework for production AI agents, organized into four categories: retrieval (context relevance, recall, precision, latency), generation (faithfulness, relevance, hallucination rate), agent behavior (tool selection accuracy, tool execution success, multi-step coherence), and production health (cost per query, P99 latency). Each metric includes measurement methodology, target thresholds, and production debugging notes drawn from 100+ enterprise deployments. The framework also covers phased implementation across pre-launch, soft launch, and stable production stages, compares existing tools like Ragas, TruLens, DeepEval, and LangSmith, and addresses common pitfalls such as using the same model for generation and judging.