Everyone wants to blame the model. “GPT-4 is hallucinating.” “Claude isn’t following instructions.” “The embeddings are bad.”
Nine times out of ten, it’s the chunks.
I’ve been auditing RAG pipelines lately and the same patterns keep showing up:
Fixed-size chunking that splits sentences in half. The model gets a fragment that starts with “…and therefore the policy applies only when” and you wonder why the answer is confidently wrong.
No overlap between chunks. Context that spans a paragraph boundary is gone. The model sees the question but not the answer, because the answer lives in the chunk after the one that matched.
Chunking before cleaning. Headers, nav menus, footer junk, and “Click here to subscribe” all get embedded as if they’re content. Then they show up in retrieval and pollute the context window.
Embedding the wrong thing. People embed the raw chunk but the user query is phrased completely differently. No wonder cosine similarity is low — they’re not semantically close, they just happen to be about the same topic.
Before you blame the model, look at what you’re actually retrieving. Log the top 5 chunks for 10 real user queries and read them like a human. If you can’t answer the question from those chunks, the model can’t either.
The model is a mirror. It reflects the quality of what you feed it.

Luis Cruz

Common RAG pipeline failures trace back to poor chunking strategies rather than model limitations. Key issues include fixed-size chunking that splits sentences mid-thought, missing overlap between chunks causing context loss at boundaries, embedding noisy content like nav menus and footers, and semantic mismatch between embedded chunks and user query phrasing. A practical diagnostic: log the top 5 retrieved chunks for real queries and assess whether a human could answer from them. If not, the model won't either.

Your RAG isn't broken. Your chunks are.

That’s why currently both the RAG is getting smarter, spawning separate agents for topic analysis to insert chunk breaks only on completed topics and injecting context references, and, also, memory systems like Claude Memory or MemPalace are storing complete non-split documents.