Choosing evaluation metrics for LLM applications starts with naming the failure mode, then picking heuristic metrics (free, deterministic, e.g. IsJson, RegexMatch, Equals, Contains, LevenshteinRatio) for rule-based checks, or LLM-as-a-judge metrics (Hallucination, AnswerRelevance, ContextRecall, ContextPrecision, Moderation) for qualitative judgments that require a model call. Opik ships 22 heuristic and 25 judge metric classes as of August 20, 2026. Custom scorers pay off when domain-specific correctness, labeled data, or a narrower judgment than built-ins provide is needed. At production volume, judge metrics compound in cost (e.g. 10,000 daily traces with one judge metric becomes ~300,000 model calls a month), so sampling, gating judges behind heuristics, running fixed evaluation sets, or using a smaller judge model are recommended. A starter set of IsJson, Hallucination, and AnswerRelevance covers most applications, with ContextRecall, ContextPrecision, and Moderation added as needs grow.
Table of contents
What failure are you trying to catch?When should you use heuristic metrics?When should you use LLM-as-a-judge metrics?How do you run these metrics in Opik?Get Brian Wones ’s stories in your inboxWhen does a custom scorer earn its keep?What do judge metrics cost at production volume?Which metrics should a team start with?Questions this post answers
When should I use a heuristic metric versus an LLM-as-a-judge metric for evaluating my LLM application?
Use a heuristic metric when the correct answer is unambiguous and expressible as a rule, since these run locally, cost nothing, and are safe to run on all traffic. Use an LLM-as-a-judge metric when the failure only shows up on reading, such as hallucination, relevance, or moderation, since these require a separate model call per item and add cost and latency. daily.dev surfaces practical evaluation guidance for teams deciding how to budget LLM judge calls.
How much does running an LLM-as-a-judge metric like Hallucination cost at production scale?
Scoring 10,000 traces a day with a single judge metric adds about 300,000 model calls a month, roughly 360 million input tokens and 30 million output tokens, priced against a provider's rate card. Approaches to control this include sampling a percentage of traffic, gating the judge behind a cheap heuristic check, running a fixed evaluation set per release, or using a smaller judge model. Engineers budgeting LLM evaluation costs can track cost-control patterns like these via daily.dev.
What metrics does Opik provide for evaluating RAG applications and what does each one detect?
Opik ships Hallucination, ContextRecall, and ContextPrecision as complementary RAG metrics: ContextRecall flags when the retriever failed to surface needed material, ContextPrecision flags when the answer drifts from correct even with context present, and Hallucination flags when the model ignores context it was given. Running all three together as of August 20, 2026 pinpoints which pipeline stage is failing. Teams diagnosing RAG failures can follow metric breakdowns like this through daily.dev.