Podium, a small-business communication platform, used LangSmith to build testing and observability pipelines for its AI Employee agent. Through dataset curation, model distillation, and pairwise evaluations, they raised the agent's F1 score from 91.7% to 98.6% and, by giving non-engineering support staff access to LangSmith traces, cut the need for engineering intervention on agent issues by 90%.
Table of contents
About PodiumTesting across the agentic development lifecycleDataset curation and fine-tuning agents with LangSmithHigh-quality customer support for AI platform without engineering interventionWhat’s Next for PodiumQuestions this post answers
How much did fine-tuning improve Podium's AI agent's ability to detect when a conversation ended?
Fine-tuning improved the F1 score for detecting natural conversation endings from 91.7% to 98.6%, a 7.5% improvement that exceeded Podium's quality threshold of 98%. The team used model distillation, curating outputs from a larger model into a smaller one, and validated the improvement with pairwise evaluations comparing the fine-tuned model against the original larger model. See how teams track agent quality improvements like this one on daily.dev.
How can non-engineers troubleshoot issues with an AI agent without escalating every problem to engineering?
Podium categorizes AI agent issues into four types: application bugs, incomplete context, misaligned instructions, and LLM errors. Only application bugs and LLM issues require engineering intervention, while incomplete context and misaligned instructions can be fixed directly by a non-engineering support team using tracing tools, cutting engineering intervention needs by 90%. daily.dev helps teams follow real-world patterns for scaling AI agent support.