--- title: Building Trustworthy Production RAG Systems Through... description: A practical guide to building a continuous evaluation pipeline for production RAG systems. Covers creating a golden dataset with diverse question categories... canonical: https://daily.dev/posts/building-trustworthy-production-rag-systems-through-continuous-evaluation-danvcfthd twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Building Trustworthy Production RAG Systems Through Continuous Evaluation | daily.dev og:description: A practical guide to building a continuous evaluation pipeline for production RAG systems. Covers creating a golden dataset with diverse question categories... og:url: https://daily.dev/posts/building-trustworthy-production-rag-systems-through-continuous-evaluation-danvcfthd og:image: https://api.daily.dev/og/posts/daNVCfTHD.png og:image:alt: Building Trustworthy Production RAG Systems Through Continuous Evaluation og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Building Trustworthy Production RAG Systems Through Continuous Evaluation **[Towards Data Science](https://daily.dev/sources/tds)** · 10 min read · 1 upvotes · 1 comments ## Summary A practical guide to building a continuous evaluation pipeline for production RAG systems. Covers creating a golden dataset with diverse question categories (multi-hop, no-answer, conflicting docs, adversarial phrasing), running manual checks before automation, scoring with RAGAS metrics (context precision, recall, faithfulness, answer relevancy), adding a custom LLM judge for domain-specific checks RAGAS can't handle, incorporating human review for disagreements, monitoring for corpus drift via production traffic sampling, and wiring everything into CI with regression thresholds to block bad merges. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does a high faithfulness score in RAGAS not mean my RAG answer is correct? Faithfulness in RAGAS only measures whether the generated answer is grounded in the retrieved context, not whether that context was the right thing to retrieve. A stale or outdated document, faithfully summarized, produces an answer that scores as fully grounded while being fully wrong, which is why faithfulness catches hallucination but not a confidently wrong source. _daily.dev surfaces practical writeups like this for teams hardening RAG evaluation beyond default metrics._ ### How many questions do I need in a golden dataset to start evaluating a RAG pipeline? Twenty to thirty questions is enough to get real signal at the start, but the category mix matters more than the raw count. Useful categories include multi-hop questions needing multiple chunks, no-answer-expected cases, conflicting or stale document scenarios, and adversarial phrasing, since a dataset of only easy factual lookups will pass everything and reveal nothing about actual failure modes. _developers designing RAG test sets can track dataset-design practices like this on daily.dev._ ### When should I use a custom LLM judge instead of RAGAS for evaluating RAG answers? A custom LLM judge should be scoped to categories where RAGAS's default prompts fall short, such as conflicting or outdated documents and cases where no answer is expected, since running a custom judge on the full dataset is slower and more expensive than RAGAS for questions it already scores reliably. The judge can check domain-specific criteria like numeric accuracy or whether the answer flags source recency instead of stating it as current fact. _teams weighing evaluation tooling tradeoffs for RAG systems can follow guidance like this on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@kartiknvj** · 0 upvotes > Continuous eval is the piece most RAG write-ups skip, so I'm glad this leads with it. Offline scores drift fast once real queries hit retrieval, and the only thing that caught it for me was scoring live traffic on groundedness. Are you sampling production traces for the eval set, or holding to a curated golden set? ## Similar posts on daily.dev - [Synthetic data for RAG evaluation: Why your RAG system needs better testing](https://daily.dev/posts/synthetic-data-for-rag-evaluation-why-your-rag-system-needs-better-testing-lo2zr1hqv) · Red Hat Developer · 1 upvotes · 0 comments - [Six Lessons Learned Building RAG Systems in Production](https://daily.dev/posts/six-lessons-learned-building-rag-systems-in-production-ou5lkxxla) · Towards Data Science · 1 upvotes · 0 comments - [Debugging Our Docs RAG, Part 1: Evaluating a Production RAG System](https://daily.dev/posts/debugging-our-docs-rag-part-1-evaluating-a-production-rag-system-nspolfjkp) · dltHub · 1 upvotes · 0 comments - [Building a Real-World RAG System: Lessons No One Tells You](https://daily.dev/posts/building-a-real-world-rag-system-lessons-no-one-tells-you-hk26kzjti) · Medium · 2 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#rag](https://daily.dev/tags/rag) [View this post on daily.dev](https://daily.dev/posts/building-trustworthy-production-rag-systems-through-continuous-evaluation-danvcfthd) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Building Trustworthy Production RAG Systems Through Continuous Evaluation","url":"https://daily.dev/posts/building-trustworthy-production-rag-systems-through-continuous-evaluation-danvcfthd","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/building-trustworthy-production-rag-systems-through-continuous-evaluation-danvcfthd"},"datePublished":"2026-07-15T15:21:01.697Z","dateModified":"2026-09-14T06:24:52.464Z","description":"A practical guide to building a continuous evaluation pipeline for production RAG systems. Covers creating a golden dataset with diverse question categories...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f7a15fa76ba9952d0a64ba5b8b1427fe?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f7a15fa76ba9952d0a64ba5b8b1427fe?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Towards Data Science","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Towards Data Science","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/tds","url":"https://daily.dev/sources/tds"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/building-trustworthy-production-rag-systems-through-continuous-evaluation-danvcfthd","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"llm,rag","timeRequired":"PT10M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Towards Data Science","item":"https://daily.dev/sources/tds"},{"@type":"ListItem","position":3,"name":"Building Trustworthy Production RAG Systems Through Continuous Evaluation"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/building-trustworthy-production-rag-systems-through-continuous-evaluation-danvcfthd","comment":[{"@type":"Comment","text":"Continuous eval is the piece most RAG write-ups skip, so I’m glad this leads with it. Offline scores drift fast once real queries hit retrieval, and the only thing that caught it for me was scoring live traffic on groundedness. Are you sampling production traces for the eval set, or holding to a curated golden set?","datePublished":"2026-07-15T18:10:25.953Z","url":"https://daily.dev/posts/daNVCfTHD#c-N0eeM5WKw","author":{"@type":"Person","name":"kartik-nvjk","url":"https://daily.dev/kartiknvj","image":"https://media.daily.dev/image/upload/s--3gGgsVCw--/f_auto/v1781456774/avatars/avatar_TvTVeiMdkRCqWUDullFmy?_a=BAMAMiWQ0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/building-trustworthy-production-rag-systems-through-continuous-evaluation-danvcfthd#faq","mainEntity":[{"@type":"Question","name":"Why does a high faithfulness score in RAGAS not mean my RAG answer is correct?","acceptedAnswer":{"@type":"Answer","text":"Faithfulness in RAGAS only measures whether the generated answer is grounded in the retrieved context, not whether that context was the right thing to retrieve. A stale or outdated document, faithfully summarized, produces an answer that scores as fully grounded while being fully wrong, which is why faithfulness catches hallucination but not a confidently wrong source. daily.dev surfaces practical writeups like this for teams hardening RAG evaluation beyond default metrics."}},{"@type":"Question","name":"How many questions do I need in a golden dataset to start evaluating a RAG pipeline?","acceptedAnswer":{"@type":"Answer","text":"Twenty to thirty questions is enough to get real signal at the start, but the category mix matters more than the raw count. Useful categories include multi-hop questions needing multiple chunks, no-answer-expected cases, conflicting or stale document scenarios, and adversarial phrasing, since a dataset of only easy factual lookups will pass everything and reveal nothing about actual failure modes. developers designing RAG test sets can track dataset-design practices like this on daily.dev."}},{"@type":"Question","name":"When should I use a custom LLM judge instead of RAGAS for evaluating RAG answers?","acceptedAnswer":{"@type":"Answer","text":"A custom LLM judge should be scoped to categories where RAGAS's default prompts fall short, such as conflicting or outdated documents and cases where no answer is expected, since running a custom judge on the full dataset is slower and more expensive than RAGAS for questions it already scores reliably. The judge can check domain-specific criteria like numeric accuracy or whether the answer flags source recency instead of stating it as current fact. teams weighing evaluation tooling tradeoffs for RAG systems can follow guidance like this on daily.dev."}}]} ```