--- title: My PDF QA Bot Cited Page 47 of a 30-Page Contract. The... description: A deep-dive into why PDF QA bots fail in production and how to build one correctly with evaluation baked in from the start. The core argument is that... canonical: https://daily.dev/posts/my-pdf-qa-bot-cited-page-47-of-a-30-page-contract-the-retriever-was-the-bug--mv61vscsq twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: My PDF QA Bot Cited Page 47 of a 30-Page Contract. The Retriever Was the Bug. | daily.dev og:description: A deep-dive into why PDF QA bots fail in production and how to build one correctly with evaluation baked in from the start. The core argument is that... og:url: https://daily.dev/posts/my-pdf-qa-bot-cited-page-47-of-a-30-page-contract-the-retriever-was-the-bug--mv61vscsq og:image: https://api.daily.dev/og/posts/MV61VsCSq.png og:image:alt: My PDF QA Bot Cited Page 47 of a 30-Page Contract. The Retriever Was the Bug. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # My PDF QA Bot Cited Page 47 of a 30-Page Contract. The Retriever Was the Bug. **[kartik-nvjk](https://daily.dev/sources/tvtveimdkrcqwudullfmy)** · [@kartiknvj](https://daily.dev/kartiknvj) · 9 min read · 0 upvotes · 0 comments ## Summary A deep-dive into why PDF QA bots fail in production and how to build one correctly with evaluation baked in from the start. The core argument is that retrieval-level evaluation is the right primitive — not generation evaluation alone — because a grounded answer on a wrong chunk still scores as passing. The post walks through seven steps: structure-preserving PDF parsing, semantic chunking on section boundaries, hybrid retrieval (vector + keyword with reciprocal rank fusion), citation enforcement with verbatim span validation, splitting eval rubrics by retrieval and generation layers, bridging those rubrics to production traces via OpenTelemetry, and closing the feedback loop by promoting failing traces into the eval dataset. Key tradeoffs include hybrid retrieval adding 5–10 points on identifier-heavy queries, semantic chunking beating fixed-character chunking by 5–15 points on retrieval rubrics, and citation enforcement adding 10–20% latency. ## Content The bot answered the easy questions and then confidently cited page 47 of a 30-page contract, returned the indemnity clause when asked about termination, and told a user the document said something it did not. The standup blamed the model. The trace said otherwise: the retriever surfaced the wrong chunk, and the model dutifully grounded its answer in it. That is the failure most PDF QA tutorials skip. A query call returns something, and without retrieval and generation evals running on every change you cannot tell whether the something is right, or whether the regression you just shipped is the chunker, the retriever, the prompt, or the model. The lesson I keep relearning is that[ retrieval-level evaluation is the right primitive for a PDF QA bot](https://futureagi.com/blog/pdf-qa-chatbot-build-evaluate-2026/?utm_source=fagiMedium&utm_medium=organic&utm_campaign=blogdistribution), not generation evaluation alone. Generation rubrics reward grounding to whatever context the retriever fed in, so if the retriever surfaced the wrong chunk, a grounded answer is still wrong and the score says it passed. The retriever is the chatbot. The generator is how the retriever talks. Here is the build with the eval baked in from step one. ## **Why most PDF QA bots ship broken** Three failure modes show up in production traces almost every time, and none of them surfaces if the eval only runs faithfulness on the final answer. - The wrong chunk gets retrieved. The user asks about Section 12, vector similarity surfaces Section 9, the generator grounds in Section 9, and groundedness passes. - The right chunks are retrieved and the model uses the wrong ones. The relevant chunks come back at ranks 1 and 4, the generator latches onto rank 1 and ignores rank 4, the answer omits a critical clause, and groundedness still passes. - A citation points at a chunk that does not contain the quoted span. The generator hallucinated the quote, structured output passed, schema validation passed, and the user is reading a fabricated citation that looks identical to a real one. The first two are retrieval failures wearing a generation failure's clothes. The third is a generation failure that needs a deterministic check, not a judge. The fix is to split the eval into a retrieval layer and a generation layer, gate both, and bridge both to production traces. ## **Step 1: parse with structure intact** PDFs lie about being simple text. A text PDF can carry multi-column layout, embedded tables, footnotes, sidebars, and figure captions, and a scanned PDF is an image that needs OCR. Most parsing bugs trace to lost structure: a parser that flattens a two-column page into interleaved nonsense, or drops a table into a wall of cells. Whichever parser you pick, its output has to preserve four fields on every element: page number, section heading, element type, and reading order. Those four fields are the difference between chunks the retriever can use and chunks that are noise. In regulated corpora, 20 percent of the documents showing up as image-only PDFs is common, and OCR quality sets the floor on your retrieval scores. ## **Step 2: chunk on a semantic boundary** Fixed-character chunking is the lazy default, and semantic chunking that respects document structure wins by 5 to 15 points on retrieval rubrics across the domains I have measured. The chunk is the unit of retrieval and attribution, so if the boundary cuts a clause in half, the citation will too. The chunker is a section walk: group elements by section, and start a new chunk when the section changes or the running size exceeds the embedding window. Three rules decide whether it earns its keep. - Keep tables and figure captions as their own chunks, and embed a table on its caption and header row rather than the full grid flattened into text. - Carry header context by prefixing each chunk with its section heading, so the embedding encodes "this is from the Liability section," not just the body. - Cap the chunk at the embedding window minus headroom for that header prefix. Evaluate the chunks before you embed them. The distribution of chunk sizes, the fraction that span a section boundary, and the fraction that include a table or caption are cheap to compute and predict the downstream retrieval scores tightly. ## **Step 3: hybrid retrieval, because pure vector loses** Pure vector search loses on identifier-heavy queries, and contract numbers, defined terms, clause IDs, statute references, drug names, and part numbers are exactly the queries that dominate enterprise PDF traffic. Pure keyword search loses the moment a user paraphrases. Hybrid retrieval, vector plus a keyword index fused with reciprocal rank fusion, wins both. Two things are non-negotiable on top of it. A per-tenant filter on every query, because a cross-tenant leak in a multi-customer deployment is a configuration failure, not a model one. And a query-rewrite step, which lifts recall on under-specified queries by a few points, though you should score both the rewrite and the final answer so you know which one helped. ## **Step 4: generate with citation enforcement** The generator is where hallucinations creep in even when retrieval did its job, and three controls compound. The output is a schema with a per-claim citation carrying the chunk ID and the quoted span, plus a confidence field. Each citation is validated deterministically before the answer leaves the bot: the chunk ID has to exist in the retrieved set, and the quoted span has to appear verbatim in that chunk. And there is a refusal path for when retrieval comes back empty or weak, with the prompt instructed to say so and drop its confidence. A failed citation validation does not ship. Retry once with a stricter prompt that names the failed citation, then fall back to a refusal rather than hand back a fabricated quote. Citation validity is the one check in the whole stack that does not need a judge, since a verbatim string match with a small fuzzy tolerance is enough, and the cheaper the check, the more often you can run it. ## **Step 5: split the eval suite by layer** Most teams write three rubrics on the final answer and call it eval, then regress for three quarters because they cannot tell whether the chunker, the retriever, the prompt, or the model moved. Split the rubrics by layer and the bisect becomes trivial. - Retrieval rubrics ask whether the retriever did its job: are the retrieved chunks relevant to the question, which retrieved chunks the answer actually used, and what fraction of the relevant content the answer used, which flags the generator ignoring good chunks. - Generation rubrics ask whether the generator did its job given the chunks it got: are the claims supported by the context, did the answer stay inside the context rather than importing outside knowledge, did it cover what the question asked, and are the asserted facts correct against ground truth. - The deterministic floor is citation validity: every cited span exists in the retrieved context, string match, no judge. Three habits separate a real gate from theatre. Set per-rubric thresholds, and fail a change on a two-point drop from the trailing seven-day baseline while an absolute floor catches the catastrophic drops. Scope by route, so a change to the contract-bot prompt does not rerun the SOP-bot suite. And diff against a moving baseline, because the models drift and the baseline has to drift with them. For the dataset, 100 to 200 cases sampled from real user questions covers factual lookup, multi-hop reasoning, table extraction, refusal scenarios, ambiguous queries where the right move is to ask for clarification, and edge cases like footnotes, scanned pages, and captions. Grow it weekly by promoting failing production traces. ## **Step 6: bridge the same rubrics to production** The gate catches the regressions you can imagine, and production catches everything else, so the same rubrics run as span-attached scores against live traces. An OpenTelemetry tracing layer with a first-class retriever span is what makes per-retriever scoring practical, since the retrieved chunks, the generation, and the verdict then live on one trace tree. Sample 5 to 10 percent of production traffic for online scoring and alarm on a two-to-five-point sustained drop in the rolling mean per rubric per route. The gap between the offline pass and the online rolling mean is its own signal: it tells you how representative the dataset still is, and a widening gap is the cue to promote more production traces into it. ## **Step 7: close the loop** The loop is what makes the work compound. Without it, every incident produces a one-off fix and the team writes the same regression twice. Cluster the failing traces by the embedding of their content into named issues, have something read each cluster and write the root cause and a concrete fix, and promote representative traces from each issue into the eval set under review. The next change touching that path has to clear the new entries, so the dataset ratchets stronger and the gate catches more each quarter. ## **Three tradeoffs I made on purpose** - Splitting retrieval and generation rubrics costs eval budget, since seven rubrics per run is more judge calls than three on the final answer. The payoff is debuggable regressions, and a classifier-backed cheap tier keeps a weekly full rerun affordable. - Hybrid retrieval adds setup, a keyword index alongside the vectors and a fusion step. The lift is 5 to 10 points on identifier-heavy queries, and a new deployment can start vector-only and add the keyword index once production traces show the failure mode. - Citation enforcement adds roughly 10 to 20 percent latency over freeform, for structured output plus per-claim validation. Worth it in compliance-sensitive domains, optional for casual ones. ## **What this comes down to** A PDF QA bot that demos well still ships broken, because the retriever is the bot and generation rubrics cannot see a retrieval bug. Parse with structure intact, chunk on semantic boundaries, retrieve hybrid with a per-tenant filter, enforce citations and validate them verbatim, and split the eval into a retrieval layer and a generation layer so you can tell which one moved. Gate both on every change, bridge both to production traces, and let the failures rebuild the dataset every week. The build is the easy part. The eval discipline is what decides whether the bot is shippable. If you have shipped a PDF QA bot, I would like to know which layer bit you first. For me it is almost always the chunker, splitting a clause across two chunks so neither one supports the answer on its own. ## Similar posts on daily.dev - [Why retrieval quality is becoming the defining challenge in AI agent architecture](https://daily.dev/posts/why-retrieval-quality-is-becoming-the-defining-challenge-in-ai-agent-architecture-jfxpaa9qo) · Vespa Blog · 0 upvotes · 0 comments - [10 Common RAG Mistakes We Keep Seeing in Production](https://daily.dev/posts/10-common-rag-mistakes-we-keep-seeing-in-production-dfkintj3j) · Towards Data Science · 1 upvotes · 0 comments - [QCon London 2026: Reliable Retrieval for Production AI Systems](https://daily.dev/posts/qcon-london-2026-reliable-retrieval-for-production-ai-systems-tdmxzfrc2) · InfoQ · 2 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#bots](https://daily.dev/tags/bots), [#rag](https://daily.dev/tags/rag), [#vector-search](https://daily.dev/tags/vector-search) [View this post on daily.dev](https://daily.dev/posts/my-pdf-qa-bot-cited-page-47-of-a-30-page-contract-the-retriever-was-the-bug--mv61vscsq) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/my-pdf-qa-bot-cited-page-47-of-a-30-page-contract-the-retriever-was-the-bug--mv61vscsq","headline":"My PDF QA Bot Cited Page 47 of a 30-Page Contract. The Retriever Was the Bug.","text":"A deep-dive into why PDF QA bots fail in production and how to build one correctly with evaluation baked in from the start. The core argument is that retrieval-level evaluation is the right primitive — not generation evaluation alone — because a grounded answer on a wrong chunk still scores as passing. The post walks through seven steps: structure-preserving PDF parsing, semantic chunking on section boundaries, hybrid retrieval (vector + keyword with reciprocal rank fusion), citation enforcement with verbatim span validation, splitting eval rubrics by retrieval and generation layers, bridging those rubrics to production traces via OpenTelemetry, and closing the feedback loop by promoting failing traces into the eval dataset. Key tradeoffs include hybrid retrieval adding 5–10 points on identifier-heavy queries, semantic chunking beating fixed-character chunking by 5–15 points on retrieval rubrics, and citation enforcement adding 10–20% latency.","url":"https://daily.dev/posts/my-pdf-qa-bot-cited-page-47-of-a-30-page-contract-the-retriever-was-the-bug--mv61vscsq","datePublished":"2026-08-15T13:12:00.142Z","dateModified":"2026-08-15T13:12:00.754Z","author":{"@type":"Person","name":"kartik-nvjk","url":"https://daily.dev/kartiknvj","image":"https://media.daily.dev/image/upload/s--3gGgsVCw--/f_auto/v1781456774/avatars/avatar_TvTVeiMdkRCqWUDullFmy?_a=BAMAMiWQ0","description":"AI Developer | Making AI reliable, trustworthy & accessible to everyone | community contributor","interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":2660}},"image":"https://media.daily.dev/image/upload/s--CBC82pYd--/f_auto/v1786468385/posts/MV61VsCSq?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/tvtveimdkrcqwudullfmy","name":"kartik-nvjk"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"kartik-nvjk","item":"https://daily.dev/sources/tvtveimdkrcqwudullfmy"},{"@type":"ListItem","position":3,"name":"My PDF QA Bot Cited Page 47 of a 30-Page Contract. The Retriever Was the Bug."}]} ```