KV caching stores key and value tensors from transformer attention layers so they can be reused during token generation instead of being recomputed. This reduces LLM inference from quadratic to linear cost per request. The article explains why LLM decoding is memory-bound rather than compute-bound, how KV cache size grows linearly with batch size and sequence length (potentially matching model weight size), and how it differs from prompt/prefix caching. Modern serving engines like vLLM address cache management challenges through paged attention (treating GPU memory like virtual memory with block tables), continuous batching (inserting new requests as slots free up), cache quantization, eviction, and offloading. These techniques together enable cost-efficient serving of long-context models at scale.
