RAG and agentic AI costs drift because retrieval, reranking, caching, tool calls, and model routing each make local decisions with no shared control layer. A single user query can fan out into 2–4 model calls consuming thousands of tokens before generation even starts. The post identifies the main cost multipliers—over-retrieval (top-k), unnecessary reranker invocations, redundant re-embedding, and lack of semantic caching—and recommends measuring eight key per-request metrics before optimizing. Practical controls include tuning retrieval depth against real query distributions, gating rerankers behind confidence thresholds, implementing semantic caching (which can cut LLM API costs up to 68.8%), and routing simple queries to cheaper models. The core architectural recommendation is centralizing these policies in a shared router layer that enforces limits before generation, rather than relying on cross-team coordination across fragmented services.
