Stanford research found ~62% of tokens sent to AI agents on every call is repeated content, yet most inference systems recompute KV vectors from scratch each time. Prefix caching helps but has a hard ceiling: any change to the cached prefix causes a full miss, breaking RAG multi-document queries, reordered documents, and growing conversation histories. LMCache is an open-source project (10k+ stars) that disaggregates cache management into a separate process, eliminating resource contention with inference. It uses shared GPU memory, zero-copy cross-GPU sharing, and parallel multi-tier loading (GPU, CPU, SSD, remote storage). On H200 GPUs with Qwen3-235B and 50 concurrent users, it delivers 14x faster time-to-first-token and 4x faster decoding. The companion research CacheBlend (EuroSys 2025 Best Paper) solves the prefix-matching limitation by selectively recomputing only the small fraction of tokens with cross-document attention, giving 2-4x faster multi-document processing with no quality loss. LMCache ships with Prometheus/OpenTelemetry, a Kubernetes operator, and fault-tolerant failover.
