Evaluating voice agents requires assessing three distinct dimensions: execution (did the agent follow its instructions?), outcome (did the interaction achieve its goal?), and experience (was the conversation smooth for the caller?). For execution, deterministic code evaluators handle explicit rule checks like tool call order, while LLM judges handle semantic requirements like policy adherence. Outcome evaluation combines LLM judges for qualitative success with downstream business metrics like booking success rate or ticket reopen rate. Experience evaluation covers latency measurement across pipeline components (STT, inference, TTS), naturalness and clarity via audio-aware LLM judges, and conversational friction signals like repeated clarification loops or failed interruption recovery. LangSmith supports all of these through traces, annotation queues, and experiment comparison, enabling a continuous evaluation loop where changes to prompts, models, or workflows can be measured against consistent criteria across the same dataset.
