--- title: Golden search datasets are anything but | daily.dev description: Golden search datasets and NDCG scores can mislead teams about real search quality, Doug Turnbull argues. A query can hit NDCG=1 even though the corpus has... canonical: https://daily.dev/posts/golden-search-datasets-are-anything-but-1xqxlqcmn twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Golden search datasets are anything but | daily.dev og:description: Golden search datasets and NDCG scores can mislead teams about real search quality, Doug Turnbull argues. A query can hit NDCG=1 even though the corpus has... og:url: https://daily.dev/posts/golden-search-datasets-are-anything-but-1xqxlqcmn og:image: https://api.daily.dev/og/posts/1XqxLqcMn.png og:image:alt: Golden search datasets are anything but og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Golden search datasets are anything but **[Software Doug](https://daily.dev/sources/softwaredoug)** · 14 min read · 0 upvotes · 0 comments ## Summary Golden search datasets and NDCG scores can mislead teams about real search quality, Doug Turnbull argues. A query can hit NDCG=1 even though the corpus has nothing truly relevant, as shown with a 'star wars rug' query on the Wayfair WANDS dataset. He proposes NDCG-m, which fixes the ideal score to the same max-grade vector for every query, giving WANDS a max NDCG-m@10 of about 0.77 versus 0.93 for Amazon's ESCI dataset. He also proposes NDCG-f, which filters out unlabeled results instead of scoring them as zero, to answer the narrower question of whether labeled relevant results rank above labeled irrelevant ones. He recommends treating NDCG as a unit test within the search team rather than a proxy for A/B test wins. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### why can my search algorithm get a perfect NDCG score but users still say the results are bad NDCG can hit 1.0 even when the underlying corpus has nothing truly relevant for a query, because the metric only measures how well you ranked the best available results, not whether those results are actually good. For example, a 'star wars rug' query against the Wayfair WANDS dataset has no results graded higher than partial relevance, so a perfect ranking of mediocre results still scores NDCG=1. _daily.dev surfaces practical evaluation pitfalls like this for teams building search ranking systems._ ### how do i handle unlabeled documents when computing NDCG for a search eval dataset Treating unlabeled results as irrelevant (grade 0) is the common research convention, but it breaks down when relevant-looking items simply weren't labeled, as happened with 'Open Box' titled products in the Amazon ESCI dataset ranking above officially labeled results. An alternative called NDCG-filtered (NDCG-f) ignores unlabeled results entirely and only scores the ordering among labeled items, answering a narrower but more reliable question about rank correctness. _developers tuning relevance pipelines can track eval design tradeoffs like this through daily.dev._ ### what is NDCG-m and how does it differ from standard NDCG NDCG-m, proposed by Max Irwin, normalizes discounted cumulative gain against a fixed ideal of all maximum-grade results (such as [3,3,3,3,3] for Amazon ESCI) instead of the best achievable ranking for each specific query. This makes the ideal score constant across queries, so NDCG-m@10 on the Wayfair WANDS dataset averages about 0.77 while Amazon ESCI reaches about 0.93, revealing how well each dataset's corpus actually serves its queries. _anyone comparing retrieval datasets can follow these nuanced metric debates via daily.dev._ ## Similar posts on daily.dev - [How to Evaluate Retrieval Quality in RAG Pipelines \(Part 3\): DCG@k and NDCG@k](https://daily.dev/posts/how-to-evaluate-retrieval-quality-in-rag-pipelines-part-3-dcg-k-and-ndcg-k-s0unw7vjd) · Towards Data Science · 0 upvotes · 0 comments - [Bag of Decisions Reranker](https://daily.dev/posts/bag-of-decisions-reranker-5j3f833zx) · Software Doug · 1 upvotes · 0 comments - [LLM-as-a-Judge: Evaluating natural language search](https://daily.dev/posts/llm-as-a-judge-evaluating-natural-language-search-qzclaql93) · Doordash · 0 upvotes · 0 comments - [Evaluate agent performance](https://daily.dev/posts/evaluate-agent-performance-znjiurbqt) · Google Cloud · 1 upvotes · 0 comments --- Tags: [#ab-testing](https://daily.dev/tags/ab-testing) [View this post on daily.dev](https://daily.dev/posts/golden-search-datasets-are-anything-but-1xqxlqcmn) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Golden search datasets are anything but","url":"https://daily.dev/posts/golden-search-datasets-are-anything-but-1xqxlqcmn","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/golden-search-datasets-are-anything-but-1xqxlqcmn"},"datePublished":"2026-10-27T20:50:25.599Z","dateModified":"2026-10-27T21:15:03.232Z","description":"Golden search datasets and NDCG scores can mislead teams about real search quality, Doug Turnbull argues. A query can hit NDCG=1 even though the corpus has...","image":"https://media.daily.dev/image/upload/s--VDukGCjf--/f_auto/v1722860399/public/Placeholder%2002","thumbnailUrl":"https://media.daily.dev/image/upload/s--VDukGCjf--/f_auto/v1722860399/public/Placeholder%2002","isAccessibleForFree":true,"articleSection":"Software Doug","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Software Doug","logo":"https://media.daily.dev/image/upload/s--i8zvexsI--/f_auto,q_auto/v1774514120/logos/softwaredoug?_a=BAMAMiWQ0","url":"https://daily.dev/sources/softwaredoug"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/golden-search-datasets-are-anything-but-1xqxlqcmn","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ab-testing","timeRequired":"PT14M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Software Doug","item":"https://daily.dev/sources/softwaredoug"},{"@type":"ListItem","position":3,"name":"Golden search datasets are anything but"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/golden-search-datasets-are-anything-but-1xqxlqcmn#faq","mainEntity":[{"@type":"Question","name":"why can my search algorithm get a perfect NDCG score but users still say the results are bad","acceptedAnswer":{"@type":"Answer","text":"NDCG can hit 1.0 even when the underlying corpus has nothing truly relevant for a query, because the metric only measures how well you ranked the best available results, not whether those results are actually good. For example, a 'star wars rug' query against the Wayfair WANDS dataset has no results graded higher than partial relevance, so a perfect ranking of mediocre results still scores NDCG=1. daily.dev surfaces practical evaluation pitfalls like this for teams building search ranking systems."}},{"@type":"Question","name":"how do i handle unlabeled documents when computing NDCG for a search eval dataset","acceptedAnswer":{"@type":"Answer","text":"Treating unlabeled results as irrelevant (grade 0) is the common research convention, but it breaks down when relevant-looking items simply weren't labeled, as happened with 'Open Box' titled products in the Amazon ESCI dataset ranking above officially labeled results. An alternative called NDCG-filtered (NDCG-f) ignores unlabeled results entirely and only scores the ordering among labeled items, answering a narrower but more reliable question about rank correctness. developers tuning relevance pipelines can track eval design tradeoffs like this through daily.dev."}},{"@type":"Question","name":"what is NDCG-m and how does it differ from standard NDCG","acceptedAnswer":{"@type":"Answer","text":"NDCG-m, proposed by Max Irwin, normalizes discounted cumulative gain against a fixed ideal of all maximum-grade results (such as [3,3,3,3,3] for Amazon ESCI) instead of the best achievable ranking for each specific query. This makes the ideal score constant across queries, so NDCG-m@10 on the Wayfair WANDS dataset averages about 0.77 while Amazon ESCI reaches about 0.93, revealing how well each dataset's corpus actually serves its queries. anyone comparing retrieval datasets can follow these nuanced metric debates via daily.dev."}}]} ```