RAG & Retrieval

Retrieval-Augmented Generation (RAG) is the practice of fetching relevant text (or other data) at query time and injecting it into the model's context so the model can answer from information it wasn't trained on. It exists because model weights are frozen at training time, context windows are finite and expensive, and hallucination drops sharply when the model is handed the actual source material. The canonical pipeline is: ingest → chunk → embed → index → retrieve → (rerank) → assemble prompt → generate, usually with a citation step so answers are auditable.

Chunking

Chunking splits documents into retrievable units. The tension is always the same. Chunks too large dilute the embedding signal and waste context budget. Chunks too small lose the surrounding meaning a passage needs to be interpretable. Common strategies:

• Fixed-size with overlap. Split every N tokens (commonly 200 to 500) with a 10 to 20% overlap so ideas straddling a boundary aren't orphaned. A simple and robust default.
• Recursive / structural. Split on natural boundaries (headings, paragraphs, sentences) before falling back to size limits. It respects document structure.
• Semantic chunking. Place boundaries where the embedding similarity between adjacent sentences drops, so each chunk is topically coherent. It costs more compute at ingest and often improves recall.
• Late chunking / contextual retrieval. Embed with document-level context in view, or prepend a short LLM-generated summary of the parent document to each chunk (Anthropic popularized "contextual retrieval," which prepends chunk-situating context before embedding). This reduces the "chunk makes no sense out of context" failure.

There is no universal best chunk size. It is an empirical knob you tune against your own eval set and your embedding model's optimal input length.

Embeddings

An embedding maps text to a dense vector such that semantically similar text lands nearby in vector space, measured by cosine similarity or dot product. Retrieval quality is bounded by embedding quality, so model choice matters. Practical considerations: dimensionality (higher can capture more nuance but costs storage and compute. Many 2026 models support Matryoshka truncation so you can trade dimensions for accuracy at query time), maximum input length (must exceed your chunk size), and domain fit (code, legal, and multilingual corpora often need specialized models). The MTEB leaderboard on Hugging Face is the standard starting point for comparing embedding models, though you should always re-benchmark on your own data. Leading options as of July 2026 include OpenAI's text-embedding-3 family, Cohere Embed, Voyage AI (now owned by MongoDB), Google's Gemini embeddings, and strong open models (Qwen3-Embedding, which sits at/near the top of MTEB, plus BGE, E5, Nomic, Jina). The same embedding model must be used for both indexing and querying.

BM25 vs vector vs hybrid

• BM25 (lexical / sparse). A decades-old bag-of-words ranking function that scores documents on exact term overlap weighted by term frequency and inverse document frequency. It works well for rare tokens, product codes, names, acronyms, and exact-match queries, but it is blind to synonyms and paraphrase.
• Vector (dense) search. Nearest-neighbor over embeddings. It captures meaning and paraphrase and can retrieve a passage that shares no words with the query, but it is weaker on exact identifiers and rare out-of-distribution terms.
• Hybrid. Run both and fuse the result lists, typically with Reciprocal Rank Fusion (RRF), which combines rankings without needing to normalize incompatible score scales. Hybrid is the common default for production RAG because lexical and semantic retrieval fail in different, complementary ways. As of 2026 this is the mainstream recommendation across most production RAG guides.

Rerankers

First-stage retrieval optimizes for recall: get the right passage somewhere in the top 50 to 100 cheaply. A reranker (usually a cross-encoder that reads the query and each candidate together, rather than comparing pre-computed vectors) then reorders those candidates for precision, so the top 3 to 8 you actually put in the prompt are the best ones. This two-stage "retrieve-then-rerank" pattern is one of the higher-leverage quality upgrades in RAG. It adds latency and per-call cost, so rerank a shortlist rather than the whole corpus. Common choices as of July 2026: Cohere Rerank, Voyage rerank, Jina reranker, and open cross-encoders like BGE reranker. ColBERT-style late-interaction models sit in between (token-level matching, more precise than single-vector, cheaper than full cross-encoding).

Retrieval evaluation

RAG has two failure surfaces, retrieval and generation, that must be evaluated separately.

• Retrieval metrics. precision@k, recall@k, MRR (mean reciprocal rank), and nDCG measure whether the right chunks were fetched and ranked highly. These need labeled query→relevant-doc pairs (a "golden set").
• Generation/answer metrics. faithfulness (is the answer grounded in the retrieved context, i.e. not hallucinated?), answer relevancy, and context precision/recall. Ragas is a widely used open-source framework for these RAG-specific metrics. DeepEval, TruLens, and Arize Phoenix are common alternatives, and general LLM-as-judge harnesses (see LLMOps: Evals, Observability & Guardrails) also apply.

Build a golden eval set of real questions early. Even 50 to 100 hand-labeled examples catch most regressions and turn "it feels better" into a number.

Common failure modes

• Bad chunking. The answer is split across two chunks, and neither retrieved chunk is sufficient.
• Retriever/generator mismatch. The right chunk is retrieved but buried at rank 40. Without reranking it never reaches the prompt.
• Lost in the middle. Models attend less to content in the middle of long contexts, so stuffing 50 chunks can perform worse than 5 well-ranked ones. Fewer, better chunks usually win.
• Stale or duplicated index. Near-duplicate chunks crowd out diverse evidence, and without a re-index pipeline answers drift from the source of truth.
• Query/document asymmetry. Short keyword queries embed poorly against long passages. Techniques like HyDE (embed a hypothetical answer) or query rewriting/expansion help.
• No citations / no grounding check. You can't tell hallucination from fact.

Frameworks & pointers

Tool	What it is
LlamaIndex	Data framework centered on RAG/indexing, with connectors and retrieval abstractions
LangChain / LangGraph	General orchestration with many retrieval integrations; LangGraph handles stateful/agentic retrieval
Haystack (deepset)	Production-oriented, pipeline-first framework
Ragas	RAG evaluation
DSPy	Programmatic prompt/pipeline optimization, increasingly used to tune RAG pipelines

Rule of thumb: reach for RAG when the knowledge is large, changes often, or must be cited. Reach for fine-tuning (Fine-Tuning & Post-Training) when you need to change behavior, format, or style, not to inject facts.