An LLM gateway acts as a single proxy endpoint between your application and multiple model providers, handling routing, caching, failover, observability, and governance. The guide compares major options (LiteLLM, OpenRouter, Portkey, Vercel AI Gateway), critically examines which cost-saving claims hold up on real traffic, and explains the mechanics of model routing, response caching, and prompt compression. Key warnings include: semantic caching carries silent false-positive risks, compressing conversation history can destroy provider prefix-cache discounts, and injecting rotating fragments (timestamps, request IDs) into system prompts is a common bug that silently busts prefix caching on every call. A decision framework helps teams choose between self-hosted and hosted options based on regulatory needs, traffic patterns, and observability requirements. An open-source CLI tool (slashspend-audit) is highlighted for measuring potential savings on your own logs before committing to any gateway.
Table of contents
Table of ContentsWhat Is an LLM Gateway?Do You Actually Need One?The Main Options ComparedWhat Gateways Actually SaveMeasure Your Own Traffic Before You CommitBuild or Buy?How to ChooseFrequently Asked QuestionsFurther ReadingQuestions this post answers
Can an LLM gateway break provider prompt caching and cost me more money?
Yes — any gateway transformation that alters the stable prefix of a request (rewriting conversation history, reordering messages, aggressive compression) invalidates the provider's prefix cache. Provider prompt caching discounts repeated prefixes by roughly 90% on input tokens, so losing it is expensive. Verify your cache-hit rate before and after adopting a gateway, and check whether the gateway rewrites history by default. Teams adopting an LLM gateway track prefix-cache hit rates and gateway behavior on daily.dev before they commit.
When does response caching in an LLM gateway actually save money vs. when is it useless?
Response caching saves the entire cost of a request when an identical (byte-identical) request has been seen recently. It pays well on classification pipelines, catalog enrichment, CI evaluation suites, scheduled jobs, and retry storms — where the same requests genuinely recur. It rarely helps in interactive chat, where phrasing varies constantly. Semantic caching raises hit rates but risks silently serving wrong answers for near-identical prompts with opposite meanings. Developers choosing between gateway caching strategies find real-traffic comparisons and edge-case discussions on daily.dev.
What is the most common hidden bug that destroys LLM prefix cache savings?
Injecting a rotating fragment — a timestamp, request ID, or session ID — into a system prompt busts the provider's prefix cache on every single call. Several popular frameworks do this by default. A 2,000-token system prompt paying full price instead of the ~90% cached rate because of an injected 'Current time:' line can dwarf every gateway optimization combined. It is free to fix and requires no gateway at all. Developers debugging unexpected LLM API bills find this class of caching pitfall covered on daily.dev.