A comprehensive handbook for building a production-grade LLM evaluation platform from scratch. Covers eval-driven development methodology, a three-tier evaluation architecture (offline dataset evaluation, CI/CD regression gates, and online production monitoring), golden dataset curation from production failures, six core RAG metrics (faithfulness, context recall, context precision, answer relevancy, hallucination, groundedness), LLM-as-judge calibration, agentic system evaluation, and automated CI/CD eval gates. Includes full working Python code and a companion GitHub repository. Uses a legal RAG application as a running example to illustrate how missing context recall metrics can cause silent production failures even when other metrics appear healthy.
Table of contents
Table of ContentsWhat You'll LearnPrerequisitesPart 1: The Eval-Driven Development ParadigmPart 2: The Three-Tier Evaluation ArchitecturePart 3: The Golden Dataset – Your Most Valuable Engineering AssetPart 4: RAG Evaluation – The Six Metrics That Carry All the Diagnostic WeightPart 5: LLM-as-Judge – How to Build an Evaluator You Can TrustPart 6: Agentic Evaluation – When the System Has Tools and MemoryPart 7: CI/CD Integration – Eval Gates That Block Bad DeploysPart 8: Production Monitoring – The Eval Loop That Never StopsPart 9: Building the Complete Eval PlatformConclusionBest Practices SummaryResourcesQuestions this post answers
What is context recall in RAG evaluation and why is it different from faithfulness?
Context recall measures whether a retriever returned all the information needed to answer a question, while faithfulness measures whether the generated answer is grounded in whatever context was retrieved. A system can score 0.95 on faithfulness while context recall sits at 0.52, meaning the answer is perfectly grounded but built from incomplete retrieved information, producing confidently wrong outputs. Teams debugging silent RAG failures compare metric tradeoffs and platform writeups on daily.dev.
How do you calibrate an LLM-as-judge evaluator before trusting it in a CI pipeline?
Collect 50 to 100 human-labelled examples spanning the quality spectrum (roughly 10 excellent, 10 poor, 30 ambiguous), run the judge on all of them, and compute Spearman's rank correlation against human scores. A correlation above 0.80 with a p-value below 0.05 is considered production-ready; also check mean absolute error and systematic bias before letting the judge block merges. Developers building trustworthy AI evaluation pipelines track calibration techniques on daily.dev.
What is the recommended sample rate and model choice for cost-effective production LLM monitoring?
Sample around 5 to 10 percent of production traces rather than evaluating every one, use gpt-4o-mini for most metrics, and reserve a stronger model like gpt-4o specifically for hallucination detection since it requires deeper factual reasoning. This combination keeps LLM-judged evaluation costs manageable, roughly $0.001 to $0.003 per test case, while preserving diagnostic power. Engineers balancing AI evaluation costs against coverage follow practical cost strategies on daily.dev.
Share this post