Daily Dose of Data Science | Avi Chawla | Substack
Read post

Rethinking KV Caching For Production Inference

Stanford research found ~62% of tokens sent to AI agents on every call is repeated content, yet most inference systems recompute KV vectors from scratch each time. Prefix caching helps but has a hard ceiling: any change to the cached prefix causes a full miss, breaking RAG multi-document queries, reordered documents, and growing conversation histories. LMCache is an open-source project (10k+ stars) that disaggregates cache management into a separate process, eliminating resource contention with inference. It uses shared GPU memory, zero-copy cross-GPU sharing, and parallel multi-tier loading (GPU, CPU, SSD, remote storage). On H200 GPUs with Qwen3-235B and 50 concurrent users, it delivers 14x faster time-to-first-token and 4x faster decoding. The companion research CacheBlend (EuroSys 2025 Best Paper) solves the prefix-matching limitation by selectively recomputing only the small fraction of tokens with cross-document attention, giving 2-4x faster multi-document processing with no quality loss. LMCache ships with Prometheus/OpenTelemetry, a Kubernetes operator, and fault-tolerant failover.

    #rag#ai-inference#vllm
Jul 07•9m read time•From blog.dailydoseofds.com
Post cover image
Table of contents
The pull request now fixes itself before anyone reads itRethinking KV caching for production inference
34 Impressions
Daily Dose of Data Science | Avi Chawla | Substack's image
Daily Dose of Data Science | Avi Chawla | Substack

Daily Dose of DS offers a daily dose of inspiration, education, and motivation for data scientists a...

1.2K Followers

•

12.8K Upvotes

Would you recommend this post?

Copy link
WhatsApp
Facebook
X
New Squad
  • © 2026 Daily Dev Ltd.
  • Guidelines
  • Explore
  • Tags
  • Sources
  • Squads
  • Leaderboard