--- title: "How do you choose evaluation metrics for an LLM application?" url: https://daily.dev/posts/how-do-you-choose-evaluation-metrics-for-an-llm-application--tv56zva90 source_url: https://heartbeat.comet.ml/how-do-you-choose-evaluation-metrics-for-an-llm-application-79f3e08e0cb0 type: article source: "HEARTBEAT" published: 2026-08-20T22:30:21.131Z updated: 2026-08-20T22:40:15.629Z tags: ["llm", "rag"] reading_time: 10 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How do you choose evaluation metrics for an LLM application? **[HEARTBEAT](https://daily.dev/sources/hrb)** · 10 min read · 0 upvotes · 0 comments ## Summary Choosing evaluation metrics for LLM applications starts with naming the failure mode, then picking heuristic metrics (free, deterministic, e.g. IsJson, RegexMatch, Equals, Contains, LevenshteinRatio) for rule-based checks, or LLM-as-a-judge metrics (Hallucination, AnswerRelevance, ContextRecall, ContextPrecision, Moderation) for qualitative judgments that require a model call. Opik ships 22 heuristic and 25 judge metric classes as of August 20, 2026. Custom scorers pay off when domain-specific correctness, labeled data, or a narrower judgment than built-ins provide is needed. At production volume, judge metrics compound in cost (e.g. 10,000 daily traces with one judge metric becomes ~300,000 model calls a month), so sampling, gating judges behind heuristics, running fixed evaluation sets, or using a smaller judge model are recommended. A starter set of IsJson, Hallucination, and AnswerRelevance covers most applications, with ContextRecall, ContextPrecision, and Moderation added as needs grow. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### When should I use a heuristic metric versus an LLM-as-a-judge metric for evaluating my LLM application? Use a heuristic metric when the correct answer is unambiguous and expressible as a rule, since these run locally, cost nothing, and are safe to run on all traffic. Use an LLM-as-a-judge metric when the failure only shows up on reading, such as hallucination, relevance, or moderation, since these require a separate model call per item and add cost and latency. _daily.dev surfaces practical evaluation guidance for teams deciding how to budget LLM judge calls._ ### How much does running an LLM-as-a-judge metric like Hallucination cost at production scale? Scoring 10,000 traces a day with a single judge metric adds about 300,000 model calls a month, roughly 360 million input tokens and 30 million output tokens, priced against a provider's rate card. Approaches to control this include sampling a percentage of traffic, gating the judge behind a cheap heuristic check, running a fixed evaluation set per release, or using a smaller judge model. _Engineers budgeting LLM evaluation costs can track cost-control patterns like these via daily.dev._ ### What metrics does Opik provide for evaluating RAG applications and what does each one detect? Opik ships Hallucination, ContextRecall, and ContextPrecision as complementary RAG metrics: ContextRecall flags when the retriever failed to surface needed material, ContextPrecision flags when the answer drifts from correct even with context present, and Hallucination flags when the model ignores context it was given. Running all three together as of August 20, 2026 pinpoints which pipeline stage is failing. _Teams diagnosing RAG failures can follow metric breakdowns like this through daily.dev._ ## Similar posts on daily.dev - [Selecting Effective LLM Evaluation Metrics: A Developer's Guide](https://daily.dev/posts/selecting-effective-llm-evaluation-metrics-a-developer-s-guide-rb4dkidmb) · PromptLayer Blog · 1 upvotes · 0 comments - [LLM evaluation metrics explained: how to measure AI system quality](https://daily.dev/posts/llm-evaluation-metrics-explained-how-to-measure-ai-system-quality-ynrvblcpe) · Netguru · 0 upvotes · 0 comments - [Best Practices and Methods for LLM Evaluation](https://daily.dev/posts/best-practices-and-methods-for-llm-evaluation-x3yfemf90) · databricks · 1 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#rag](https://daily.dev/tags/rag) [View this post on daily.dev](https://daily.dev/posts/how-do-you-choose-evaluation-metrics-for-an-llm-application--tv56zva90)