--- title: "How Do You Build an LLM Evaluation Dataset from Production Traces?" url: https://daily.dev/posts/how-do-you-build-an-llm-evaluation-dataset-from-production-traces--pmibina7y source_url: https://heartbeat.comet.ml/how-do-you-build-an-llm-evaluation-dataset-from-production-traces-2cad3ad175b6 type: article source: "HEARTBEAT" published: 2026-08-25T22:36:23.492Z updated: 2026-08-25T22:38:33.282Z tags: ["llm", "testing", "observability", "rag"] reading_time: 9 upvotes: 2 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How Do You Build an LLM Evaluation Dataset from Production Traces? **[HEARTBEAT](https://daily.dev/sources/hrb)** · 9 min read · 2 upvotes · 0 comments ## Summary A practical guide to building LLM evaluation datasets from production traces using Opik, the open-source LLM observability platform from Comet. Covers which traces to prioritize (high-volume patterns, known failures, edge cases), what fields a dataset item needs (input, expected output, metadata), three approaches for handling missing ground truth (reference-free LLM-as-a-Judge metrics, human annotation, model-generated drafts reviewed by humans), code examples for inserting items via the Opik Python SDK, the distinction between datasets/metrics and Opik Test Suites, and signals that indicate a dataset needs refreshing (prompt changes, distribution shift, metric saturation). ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What fields does a dataset item need for LLM evaluation with Opik? An Opik dataset item stores three fields: input, optional expected output, and optional metadata. Input is the exact production prompt or message list, unmodified. Expected output is required for heuristic metrics like Equals or LevenshteinRatio, but optional for LLM-as-a-Judge metrics like Hallucination or AnswerRelevance. Metadata typically holds the source trace ID, date range, user segment, and prompt version for later filtering. _daily.dev surfaces practical guides like this for teams wiring up LLM evaluation pipelines._ ### How do you create an evaluation dataset when there's no ground truth for LLM outputs? Three practical approaches exist depending on human review bandwidth: reference-free LLM-as-a-Judge metrics like Hallucination, Moderation, and AnswerRelevance that score output without an expected answer; human annotation of a representative sample routed through Opik's Annotation Queues; or generating draft expected outputs with a stronger model and having a human accept, edit, or reject each one. A hybrid of a small annotated core plus a larger reference-free set is most common. _Teams choosing between annotation strategies for AI evaluation often compare notes on daily.dev._ ### How often should an LLM evaluation dataset be refreshed to stay representative? A monthly refresh of roughly 10 to 20 percent of the dataset is a practical cadence for most teams, replacing the oldest items with recent production traces. Three signals also trigger an earlier refresh: shipping a new prompt version or retrieval strategy, a material shift in query types or user segments, and metric scores that stop moving because the dataset has saturated. _Keeping evaluation pipelines current is easier when engineers track workflow changes like this on daily.dev._ ## Similar posts on daily.dev - [​Trace and Monitor Any AI/LLM App](https://daily.dev/posts/trace-and-monitor-any-ai-llm-app-1cyxkbjw9) · Daily Dose of Data Science \| Avi Chawla \| Substack · 6 upvotes · 0 comments - [Step-by-Step Guide to Assessing LLM Observability Tools for AI Teams](https://daily.dev/posts/step-by-step-guide-to-assessing-llm-observability-tools-for-ai-teams-dujbgzbws) · PromptLayer Blog · 1 upvotes · 0 comments - [AI Evaluation Engineering: Build a Production-Grade LLM Evaluation Platform from Scratch \[Full Handbook\]](https://daily.dev/posts/ai-evaluation-engineering-build-a-production-grade-llm-evaluation-platform-from-scratch-full-handb-ju9v0rogq) · freeCodeCamp · 5 upvotes · 0 comments - [The complete guide to LLM observability for 2026](https://daily.dev/posts/the-complete-guide-to-llm-observability-for-2026-vabqmevzx) · portkey · 1 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#testing](https://daily.dev/tags/testing), [#observability](https://daily.dev/tags/observability), [#rag](https://daily.dev/tags/rag) [View this post on daily.dev](https://daily.dev/posts/how-do-you-build-an-llm-evaluation-dataset-from-production-traces--pmibina7y)