--- title: Your Retrieval Metrics Are Measured Against an... description: A hands-on experiment building a RAG evaluation harness inside OutSystems Developer Cloud compares Vector, BM25, and Hybrid search across 52 hand-written... canonical: https://daily.dev/posts/your-retrieval-metrics-are-measured-against-an-incomplete-answer-key-noroy7smo twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Your Retrieval Metrics Are Measured Against an Incomplete Answer Key | daily.dev og:description: A hands-on experiment building a RAG evaluation harness inside OutSystems Developer Cloud compares Vector, BM25, and Hybrid search across 52 hand-written... og:url: https://daily.dev/posts/your-retrieval-metrics-are-measured-against-an-incomplete-answer-key-noroy7smo og:image: https://api.daily.dev/og/posts/NOROY7SmO.png og:image:alt: Your Retrieval Metrics Are Measured Against an Incomplete Answer Key og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Your Retrieval Metrics Are Measured Against an Incomplete Answer Key **[ITNEXT](https://daily.dev/sources/itnext)** · 11 min read · 0 upvotes · 0 comments ## Summary A hands-on experiment building a RAG evaluation harness inside OutSystems Developer Cloud compares Vector, BM25, and Hybrid search across 52 hand-written questions and 1,063 human relevance grades. The headline finding: Hybrid wins on blended averages but loses that lead once questions are split by type, with BM25 beating Hybrid on exact-term lookups and losing badly on conceptual ones. The more instructive story is methodological: the answer key used to grade results was silently incomplete, missing 65% then 97% coverage of what BM25 actually returned, because the candidate pool was built as one global top-15 across methods rather than per-method. Fixing the coverage gap changed a recall figure and reinforced the need to check answer-key completeness before trusting any retrieval benchmark. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Does hybrid search always beat BM25 alone for retrieval in a RAG system? Not on every question type. In a controlled test with 52 questions across 7 documents, hybrid search led on blended Precision@1 (88.5% vs BM25's 84.6%), but BM25 alone actually beat hybrid on exact-term lookups (92.0% vs 88.0%), while hybrid tied vector search on conceptual questions and BM25 came last there at 73.9%. Hybrid never finished first in either category but also never finished last. _Developers comparing retrieval strategies for RAG pipelines can track findings like this on daily.dev._ ### Why would my retrieval evaluation scores look wrong after adding a new search method like BM25 to the comparison? Your answer key is likely missing grades for results the new method surfaces through a different matching mechanism. In one test, adding BM25 (word matching) alongside vector and hybrid search dropped grading coverage from 86% to 65%, because 64 of BM25's results had never been graded before since no prior method had returned them. Recomputing scores against the completed key changed a recall figure from 79.7% to 73.2%. _Anyone building retrieval benchmarks can follow evaluation methodology write-ups like this via daily.dev._ ### Why does BM25 coverage never reach 100% in retrieval evaluation even after fixing the answer key? BM25 only returns a chunk if it shares actual vocabulary with the query, so for 20 of 52 test questions it found fewer than 15 matching chunks in the entire corpus, sometimes only one. Since coverage is calculated by dividing by a fixed 15 results regardless of how many actually existed, BM25 settles at 85% coverage even when every found result is fully graded, versus 100% for vector and hybrid search which always pull 40 candidates. _daily.dev helps developers dig into the mechanics behind search evaluation quirks like this one._ ## Similar posts on daily.dev - [I Stopped Using Vector Databases. My RAG Got Better.](https://daily.dev/posts/i-stopped-using-vector-databases-my-rag-got-better--msjc5zpqk) · Medium · 0 upvotes · 0 comments - [Hybrid Search Explained: Combining Vector and Keyword Retrieval](https://daily.dev/posts/hybrid-search-explained-combining-vector-and-keyword-retrieval-csaektsg6) · BigData Boutique blog · 0 upvotes · 0 comments - [Hybrid Search and Re-Ranking in Production RAG](https://daily.dev/posts/hybrid-search-and-re-ranking-in-production-rag-u32wjtkwm) · Towards Data Science · 0 upvotes · 0 comments - [RAG with Hybrid Search: How Does Keyword Search Work?](https://daily.dev/posts/rag-with-hybrid-search-how-does-keyword-search-work--fd4ni5gk5) · Towards Data Science · 0 upvotes · 0 comments - [Why Vector Search Alone Isn't Enough: Hybrid Retrieval for RAG](https://daily.dev/posts/why-vector-search-alone-isn-t-enough-hybrid-retrieval-for-rag-si650vuzf) · InfoQ · 0 upvotes · 0 comments --- Tags: [#rag](https://daily.dev/tags/rag), [#vector-search](https://daily.dev/tags/vector-search) [View this post on daily.dev](https://daily.dev/posts/your-retrieval-metrics-are-measured-against-an-incomplete-answer-key-noroy7smo) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Your Retrieval Metrics Are Measured Against an Incomplete Answer Key","url":"https://daily.dev/posts/your-retrieval-metrics-are-measured-against-an-incomplete-answer-key-noroy7smo","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/your-retrieval-metrics-are-measured-against-an-incomplete-answer-key-noroy7smo"},"datePublished":"2026-09-01T08:22:37.310Z","dateModified":"2026-09-01T08:23:09.269Z","description":"A hands-on experiment building a RAG evaluation harness inside OutSystems Developer Cloud compares Vector, BM25, and Hybrid search across 52 hand-written...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/529c4d77bf47325cf0452f17ca23d654?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/529c4d77bf47325cf0452f17ca23d654?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"ITNEXT","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"ITNEXT","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/itnext2","url":"https://daily.dev/sources/itnext"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/your-retrieval-metrics-are-measured-against-an-incomplete-answer-key-noroy7smo","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"rag,vector-search","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"ITNEXT","item":"https://daily.dev/sources/itnext"},{"@type":"ListItem","position":3,"name":"Your Retrieval Metrics Are Measured Against an Incomplete Answer Key"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/your-retrieval-metrics-are-measured-against-an-incomplete-answer-key-noroy7smo#faq","mainEntity":[{"@type":"Question","name":"Does hybrid search always beat BM25 alone for retrieval in a RAG system?","acceptedAnswer":{"@type":"Answer","text":"Not on every question type. In a controlled test with 52 questions across 7 documents, hybrid search led on blended Precision@1 (88.5% vs BM25's 84.6%), but BM25 alone actually beat hybrid on exact-term lookups (92.0% vs 88.0%), while hybrid tied vector search on conceptual questions and BM25 came last there at 73.9%. Hybrid never finished first in either category but also never finished last. Developers comparing retrieval strategies for RAG pipelines can track findings like this on daily.dev."}},{"@type":"Question","name":"Why would my retrieval evaluation scores look wrong after adding a new search method like BM25 to the comparison?","acceptedAnswer":{"@type":"Answer","text":"Your answer key is likely missing grades for results the new method surfaces through a different matching mechanism. In one test, adding BM25 (word matching) alongside vector and hybrid search dropped grading coverage from 86% to 65%, because 64 of BM25's results had never been graded before since no prior method had returned them. Recomputing scores against the completed key changed a recall figure from 79.7% to 73.2%. Anyone building retrieval benchmarks can follow evaluation methodology write-ups like this via daily.dev."}},{"@type":"Question","name":"Why does BM25 coverage never reach 100% in retrieval evaluation even after fixing the answer key?","acceptedAnswer":{"@type":"Answer","text":"BM25 only returns a chunk if it shares actual vocabulary with the query, so for 20 of 52 test questions it found fewer than 15 matching chunks in the entire corpus, sometimes only one. Since coverage is calculated by dividing by a fixed 15 results regardless of how many actually existed, BM25 settles at 85% coverage even when every found result is fully graded, versus 100% for vector and hybrid search which always pull 40 candidates. daily.dev helps developers dig into the mechanics behind search evaluation quirks like this one."}}]} ```