Long-context LLM inference gets expensive primarily because of the KV cache — a block of GPU memory storing key and value vectors for every input token. During decoding, the model reads the entire cache on every generated token, making it memory-bandwidth-bound rather than compute-bound. For a 70B model at 128K tokens, this cache alone can reach ~40GB. Several optimization techniques address different parts of the cache size equation: grouped-query attention and multi-head latent attention reduce per-token footprint at the architecture level; quantization (8-bit or 4-bit) shrinks bytes per stored value; eviction drops tokens unlikely to be needed again; and paged attention plus prefix caching improve how memory is allocated and shared across requests. Each technique involves tradeoffs — grouped-query attention and paged attention are near-free wins, while eviction risks losing critical context and latent attention requires significant serving engineering.
