--- title: "LLM-as-a-Judge: Evaluating natural language search" url: https://daily.dev/posts/llm-as-a-judge-evaluating-natural-language-search-qzclaql93 source_url: https://careersatdoordash.com/blog/doordash-llm-as-a-judge-evaluating-natural-language-search type: article source: "Doordash" published: 2026-05-14T17:09:27.685Z updated: 2026-05-14T17:09:58.943Z reading_time: 19 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # LLM-as-a-Judge: Evaluating natural language search **[Doordash](https://daily.dev/sources/doordash)** · 19 min read · 0 upvotes · 0 comments ## Summary DoorDash built a natural language search (NLS) system that handles vague, multi-constraint queries like 'cozy date night dinner.' Evaluating such a system proved harder than building it — traditional metrics were noisy and human annotation was slow and inconsistent. An audit of 6,824 query-store judgments revealed that human labels were wrong in over half of reviewed cases, with disagreement concentrated at boundary relevance levels. The root cause was a rubric that forced annotators to improvise on subjective, multi-faceted queries. The solution was to decompose relevance into independent binary facets (e.g., 'Does this store serve tacos?' 'Does it offer items under $12?'), calibrate an LLM judge against adjudicated human consensus, and automate execution via daily monitoring and PR-level guardrails. Per-facet NDCG evaluation exposed hidden failures in speed and geolocation that aggregate scores masked. Key lessons include starting with binary relevance, treating the rubric as a versioned product artifact, and investing early in context completeness (item-level pricing, customization flags, display logic). ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Using LLMs to infer grocery preferences from DoorDash restaurant orders](https://daily.dev/posts/using-llms-to-infer-grocery-preferences-from-doordash-restaurant-orders-ig7ur8p29) · Doordash · 1 upvotes · 0 comments - [Search Quality Assurance with AI as a Judge](https://daily.dev/posts/search-quality-assurance-with-ai-as-a-judge-ckpklk2ko) · Zalando · 3 upvotes · 0 comments - [Using small language models to serve more relevant DoorDash search ads](https://daily.dev/posts/using-small-language-models-to-serve-more-relevant-doordash-search-ads-tmgus790f) · Doordash · 0 upvotes · 0 comments --- [View this post on daily.dev](https://daily.dev/posts/llm-as-a-judge-evaluating-natural-language-search-qzclaql93)