Production RAG systems often produce confident but wrong answers at scale because retrieval architectures designed for small demos break down with millions of documents. The core problem is recall failure: the right document never enters the candidate set, so no prompt or model improvement can fix it. A scalable solution requires treating retrieval as a unified serving system with four key principles: (1) unified retrieval instead of fragmented services, (2) hybrid search combining semantic and keyword signals with large candidate sets, (3) multi-stage ranking that applies expensive neural rerankers only to a refined subset, and (4) recognizing that retrieval quality — not model quality — determines overall system quality. Four specific failure 'cliffs' are identified: shallow candidate generation, fragmented services adding latency, overly broad reranking, and using prompt engineering as a substitute for retrieval quality.
