--- title: Why DeepSeek stores its KV cache on disk | daily.dev description: DeepSeek stores its KV cache on SSDs instead of GPU memory, and open-sourced DeepInfer, the inference stack behind it. Cache hits return the first token up to... canonical: https://daily.dev/posts/why-deepseek-stores-its-kv-cache-on-disk-pmhln2sue twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Why DeepSeek stores its KV cache on disk | daily.dev og:description: DeepSeek stores its KV cache on SSDs instead of GPU memory, and open-sourced DeepInfer, the inference stack behind it. Cache hits return the first token up to... og:url: https://daily.dev/posts/why-deepseek-stores-its-kv-cache-on-disk-pmhln2sue og:image: https://api.daily.dev/og/posts/pMhLn2Sue.png og:image:alt: Why DeepSeek stores its KV cache on disk og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Why DeepSeek stores its KV cache on disk **[Daily Dose of Data Science \| Avi Chawla \| Substack](https://daily.dev/sources/dailydoseofds)** · 9 min read · 17 upvotes · 3 comments ## Summary DeepSeek stores its KV cache on SSDs instead of GPU memory, and open-sourced DeepInfer, the inference stack behind it. Cache hits return the first token up to 26 times faster. In 2024, a 128K-token prompt that hit the cache got its first token in 500ms instead of 13s. Multi-head Latent Attention, introduced in DeepSeek-V2, cut the KV cache size by 93.3% versus DeepSeek's earlier 67B model, making disk storage viable. The cache lives on 3FS, DeepSeek's distributed file system spreading chunks across thousands of SSDs. A day of production traffic had 56% of input tokens served from disk, and a cache hit costs 50x less than a miss on deepseek-flash. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does DeepSeek store its KV cache on disk instead of GPU memory? DeepSeek stores its KV cache on SSDs because Multi-head Latent Attention (MLA), introduced in DeepSeek-V2, shrank the cache by 93.3% compared to its earlier 67B model, making disk reads faster than recomputing. The cache lives on 3FS, DeepSeek's distributed file system, which spreads chunks across thousands of SSDs so reads pull from many drives in parallel. _Anyone tuning LLM serving costs can track infrastructure techniques like this one on daily.dev._ ### How much faster is DeepSeek's cached first token response compared to a cache miss? A 128K-token prompt that hits the cache returns its first token in 500ms instead of 13 seconds without the cache, a speedup DeepSeek reported when it launched this system in 2024. Cache hits overall return the first token up to 26 times faster, and on deepseek-flash a cache hit costs 50 times less than a miss. _Developers optimizing prompt caching strategies can follow these kinds of inference benchmarks on daily.dev._ ### What is Multi-head Latent Attention (MLA) in DeepSeek's models? Multi-head Latent Attention is an attention mechanism introduced in DeepSeek-V2 that stores one small compressed vector per token per layer instead of separate key-value pairs for every attention head, rebuilding the keys and values during attention. This cut the KV cache size by 93.3% compared to DeepSeek's earlier 67B model, making disk-based caching practical. _Those comparing attention mechanisms for efficient LLM serving can follow this kind of architecture detail on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@aliahmeddiyweb** · 0 upvotes > The 93.3% KV cache reduction from Multi-head Latent Attention in DeepSeek-V2 is what makes the disk approach workable, since the cached entries become small enough to move around. A first token in 500ms instead of 13s for a 128K-token prompt is a big gap, and 56% of input tokens coming from disk shows how repetitive real prompts are. I wonder how much 3FS read latency matters for short prompts, where recomputing might be close to as fast. **@kartiknvj** · 0 upvotes > On your closing question, yes. Since only an exact match from the first token counts, I keep the system prompt and tool definitions identical at the top of every call, tools in a fixed order, and push anything that changes per call, like a timestamp, below them. With DeepInfer keeping cached prefixes 3 days for agent traffic, that stable top can keep hitting across separate sessions too. **@ahmetozel** · 0 upvotes > The shared 3FS cache removes the need to route a repeat back to its original prefill machine, but it also makes storage contention part of inference scheduling. The 500ms hit example is compelling; I would want its tail latency when many requests restore long prefixes concurrently. > > A useful benchmark would mix cache restores with misses that are writing new blocks, while keeping the token-length distribution fixed. That would show whether admission control should sometimes choose recomputation when the shared read queue is busy, even though a reusable prefix exists. ## Similar posts on daily.dev - [DeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Since Day 0](https://daily.dev/posts/deepseek-v4-1-flash-on-vllm-5x-agentic-throughput-since-day-0-js5lna7tm) · vLLM · 0 upvotes · 0 comments --- Tags: [#ai-inference](https://daily.dev/tags/ai-inference), [#deepseek](https://daily.dev/tags/deepseek) [View this post on daily.dev](https://daily.dev/posts/why-deepseek-stores-its-kv-cache-on-disk-pmhln2sue) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Why DeepSeek stores its KV cache on disk","url":"https://daily.dev/posts/why-deepseek-stores-its-kv-cache-on-disk-pmhln2sue","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/why-deepseek-stores-its-kv-cache-on-disk-pmhln2sue"},"datePublished":"2026-10-28T23:02:52.338Z","dateModified":"2026-10-28T23:56:09.597Z","description":"DeepSeek stores its KV cache on SSDs instead of GPU memory, and open-sourced DeepInfer, the inference stack behind it. Cache hits return the first token up to...","image":"https://media.daily.dev/image/fetch/s--KSFIKLg0--/f_auto,pg_1,q_auto/https://substackcdn.com/image/fetch/%24s_%21Ppcc%21%2Cw_1200%2Ch_675%2Cc_fill%2Cf_jpg%2Cq_auto:good%2Cfl_progressive:steep%2Cg_auto/https%253A%252F%252Fsubstack-post-media.s3.amazonaws.com%252Fpublic%252Fimages%252F7767340f-c861-4882-a49f-2f4f3fafcf58_1200x806.gif?_a=BAMAMiiw0","thumbnailUrl":"https://media.daily.dev/image/fetch/s--KSFIKLg0--/f_auto,pg_1,q_auto/https://substackcdn.com/image/fetch/%24s_%21Ppcc%21%2Cw_1200%2Ch_675%2Cc_fill%2Cf_jpg%2Cq_auto:good%2Cfl_progressive:steep%2Cg_auto/https%253A%252F%252Fsubstack-post-media.s3.amazonaws.com%252Fpublic%252Fimages%252F7767340f-c861-4882-a49f-2f4f3fafcf58_1200x806.gif?_a=BAMAMiiw0","isAccessibleForFree":true,"articleSection":"Daily Dose of Data Science | Avi Chawla | Substack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Daily Dose of Data Science | Avi Chawla | Substack","logo":"https://media.daily.dev/image/upload/s--4IHQgTOw--/f_auto/v1710503712/logos/dailydoseofds","url":"https://daily.dev/sources/dailydoseofds"},"commentCount":3,"discussionUrl":"https://daily.dev/posts/why-deepseek-stores-its-kv-cache-on-disk-pmhln2sue","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":17},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":3}],"keywords":"ai-inference,deepseek","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Daily Dose of Data Science | Avi Chawla | Substack","item":"https://daily.dev/sources/dailydoseofds"},{"@type":"ListItem","position":3,"name":"Why DeepSeek stores its KV cache on disk"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/why-deepseek-stores-its-kv-cache-on-disk-pmhln2sue","comment":[{"@type":"Comment","text":"The 93.3% KV cache reduction from Multi-head Latent Attention in DeepSeek-V2 is what makes the disk approach workable, since the cached entries become small enough to move around. A first token in 500ms instead of 13s for a 128K-token prompt is a big gap, and 56% of input tokens coming from disk shows how repetitive real prompts are. I wonder how much 3FS read latency matters for short prompts, where recomputing might be close to as fast.","datePublished":"2026-10-29T12:11:35.130Z","url":"https://daily.dev/posts/pMhLn2Sue#c-dyKrogi8R","author":{"@type":"Person","name":"Ali Ahmed","url":"https://daily.dev/aliahmeddiyweb","image":"https://media.daily.dev/image/upload/s--b4sqNNv5--/f_auto,pg_1/v1792538576/avatars/avatar_lsiC0qqIxPUeSVlhCejTA?_a=BAMAMicg0"}},{"@type":"Comment","text":"On your closing question, yes. Since only an exact match from the first token counts, I keep the system prompt and tool definitions identical at the top of every call, tools in a fixed order, and push anything that changes per call, like a timestamp, below them. With DeepInfer keeping cached prefixes 3 days for agent traffic, that stable top can keep hitting across separate sessions too.","datePublished":"2026-10-29T13:50:56.154Z","url":"https://daily.dev/posts/pMhLn2Sue#c-EpBVym1ch","author":{"@type":"Person","name":"kartik-nvjk","url":"https://daily.dev/kartiknvj","image":"https://media.daily.dev/image/upload/s--3gGgsVCw--/f_auto/v1781456774/avatars/avatar_TvTVeiMdkRCqWUDullFmy?_a=BAMAMiWQ0"}},{"@type":"Comment","text":"The shared 3FS cache removes the need to route a repeat back to its original prefill machine, but it also makes storage contention part of inference scheduling. The 500ms hit example is compelling; I would want its tail latency when many requests restore long prefixes concurrently.\nA useful benchmark would mix cache restores with misses that are writing new blocks, while keeping the token-length distribution fixed. That would show whether admission control should sometimes choose recomputation when the shared read queue is busy, even though a reusable prefix exists.","datePublished":"2026-10-30T05:44:26.937Z","url":"https://daily.dev/posts/pMhLn2Sue#c-FI9xK1tZ0","author":{"@type":"Person","name":"Ahmet Özel","url":"https://daily.dev/ahmetozel","image":"https://avatars.githubusercontent.com/u/70992231?v=4"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/why-deepseek-stores-its-kv-cache-on-disk-pmhln2sue#faq","mainEntity":[{"@type":"Question","name":"Why does DeepSeek store its KV cache on disk instead of GPU memory?","acceptedAnswer":{"@type":"Answer","text":"DeepSeek stores its KV cache on SSDs because Multi-head Latent Attention (MLA), introduced in DeepSeek-V2, shrank the cache by 93.3% compared to its earlier 67B model, making disk reads faster than recomputing. The cache lives on 3FS, DeepSeek's distributed file system, which spreads chunks across thousands of SSDs so reads pull from many drives in parallel. Anyone tuning LLM serving costs can track infrastructure techniques like this one on daily.dev."}},{"@type":"Question","name":"How much faster is DeepSeek's cached first token response compared to a cache miss?","acceptedAnswer":{"@type":"Answer","text":"A 128K-token prompt that hits the cache returns its first token in 500ms instead of 13 seconds without the cache, a speedup DeepSeek reported when it launched this system in 2024. Cache hits overall return the first token up to 26 times faster, and on deepseek-flash a cache hit costs 50 times less than a miss. Developers optimizing prompt caching strategies can follow these kinds of inference benchmarks on daily.dev."}},{"@type":"Question","name":"What is Multi-head Latent Attention (MLA) in DeepSeek's models?","acceptedAnswer":{"@type":"Answer","text":"Multi-head Latent Attention is an attention mechanism introduced in DeepSeek-V2 that stores one small compressed vector per token per layer instead of separate key-value pairs for every attention head, rebuilding the keys and values during attention. This cut the KV cache size by 93.3% compared to DeepSeek's earlier 67B model, making disk-based caching practical. Those comparing attention mechanisms for efficient LLM serving can follow this kind of architecture detail on daily.dev."}}]} ```