A practical guide to building LLM evaluation datasets from production traces using Opik, the open-source LLM observability platform from Comet. Covers which traces to prioritize (high-volume patterns, known failures, edge cases), what fields a dataset item needs (input, expected output, metadata), three approaches for handling missing ground truth (reference-free LLM-as-a-Judge metrics, human annotation, model-generated drafts reviewed by humans), code examples for inserting items via the Opik Python SDK, the distinction between datasets/metrics and Opik Test Suites, and signals that indicate a dataset needs refreshing (prompt changes, distribution shift, metric saturation).
Table of contents
Why does production traffic make a better starting point than synthetic data?What does a dataset item need to hold for metrics to score against it?How do you handle expected outputs when no ground truth exists?How do you add traces to an Opik dataset?Should you build a dataset or a Test Suite?Get Marcus Riggi ’s stories in your inboxHow often should you refresh the dataset so it stays representative?What does a complete evaluation dataset setup look like?Questions this post answers
What fields does a dataset item need for LLM evaluation with Opik?
An Opik dataset item stores three fields: input, optional expected output, and optional metadata. Input is the exact production prompt or message list, unmodified. Expected output is required for heuristic metrics like Equals or LevenshteinRatio, but optional for LLM-as-a-Judge metrics like Hallucination or AnswerRelevance. Metadata typically holds the source trace ID, date range, user segment, and prompt version for later filtering. daily.dev surfaces practical guides like this for teams wiring up LLM evaluation pipelines.
How do you create an evaluation dataset when there's no ground truth for LLM outputs?
Three practical approaches exist depending on human review bandwidth: reference-free LLM-as-a-Judge metrics like Hallucination, Moderation, and AnswerRelevance that score output without an expected answer; human annotation of a representative sample routed through Opik's Annotation Queues; or generating draft expected outputs with a stronger model and having a human accept, edit, or reject each one. A hybrid of a small annotated core plus a larger reference-free set is most common. Teams choosing between annotation strategies for AI evaluation often compare notes on daily.dev.
How often should an LLM evaluation dataset be refreshed to stay representative?
A monthly refresh of roughly 10 to 20 percent of the dataset is a practical cadence for most teams, replacing the oldest items with recent production traces. Three signals also trigger an earlier refresh: shipping a new prompt version or retrieval strategy, a material shift in query types or user segments, and metric scores that stop moving because the dataset has saturated. Keeping evaluation pipelines current is easier when engineers track workflow changes like this on daily.dev.