--- title: Why KV Cache Stores K and V Vectors But Never Q? | daily.dev description: A deep dive explains why the KV cache in LLM inference stores only key and value vectors, never query vectors. Walking through prefill and decoding steps token... canonical: https://daily.dev/posts/why-kv-cache-stores-k-and-v-vectors-but-never-q--gz6gkpinu twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Why KV Cache Stores K and V Vectors But Never Q? | daily.dev og:description: A deep dive explains why the KV cache in LLM inference stores only key and value vectors, never query vectors. Walking through prefill and decoding steps token... og:url: https://daily.dev/posts/why-kv-cache-stores-k-and-v-vectors-but-never-q--gz6gkpinu og:image: https://api.daily.dev/og/posts/gz6gKpinU.png og:image:alt: Why KV Cache Stores K and V Vectors But Never Q? og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Why KV Cache Stores K and V Vectors But Never Q? **[Daily Dose of Data Science \| Avi Chawla \| Substack](https://daily.dev/sources/dailydoseofds)** · 7 min read · 1 upvotes · 0 comments ## Summary A deep dive explains why the KV cache in LLM inference stores only key and value vectors, never query vectors. Walking through prefill and decoding steps token by token, it shows that each decode step only needs the current position's query (used once and discarded) while keys and values from all prior positions must be retained since they never change under causal masking. It also covers Datalab's Marker v2, an open-source PDF/document parsing pipeline using a shared 650M-parameter Surya 2 model server architecture that boosts GPU throughput (2.9-23.7 pages/sec on a B200) by batching pages across many lightweight CPU workers instead of running isolated model copies per worker. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does the KV cache in transformer models store key and value vectors but not the query vector? Each decode step only needs the query vector for the newest token position to compute that step's attention output, and it is never reused afterward, so caching it wastes memory. Keys and values, however, depend only on a token and the tokens before it under causal masking, so once computed they stay valid for every future decoding step and must be retained to avoid recomputation. _daily.dev surfaces deep dives like this for engineers optimizing LLM inference and serving costs._ ### What causes the first token in LLM generation to take longer than subsequent tokens? The delay comes from prefill, the compute-heavy forward pass where all prompt tokens are processed in parallel with causal masking to produce query, key, and value vectors at every layer for every position. Because prefill computes attention across the entire prompt at once before any output token is generated, it dominates time-to-first-token (TTFT), while later tokens only require processing one new position. _understanding prefill versus decode helps teams reasoning about LLM latency budgets on daily.dev._ ### How fast is Datalab's Marker v2 PDF parsing pipeline and what model does it use? Marker v2 uses a single 650-million-parameter model called Surya 2 that handles OCR, layout, reading order, and table recognition across more than 90 languages. On a single B200 GPU it sustains 2.9 pages per second in balanced mode, 7.4 in fast mode, and 23.7 with OCR disabled, by routing many lightweight CPU workers to one shared inference server instead of giving each worker its own model copy. _developers comparing document-parsing pipelines can track benchmarks like this on daily.dev._ ## Similar posts on daily.dev - [How KV Caching Slashes LLM Inference Costs at Scale](https://daily.dev/posts/how-kv-caching-slashes-llm-inference-costs-at-scale-7g9k0lnod) · DigitalOcean Community · 0 upvotes · 0 comments - [KV Caching in LLMs: A Guide for Developers](https://daily.dev/posts/kv-caching-in-llms-a-guide-for-developers-hfpim4eo0) · Machine Learning Mastery · 1 upvotes · 0 comments - [From Prompt to Prediction: Understanding Prefill, Decode, and the KV Cache in LLMs](https://daily.dev/posts/from-prompt-to-prediction-understanding-prefill-decode-and-the-kv-cache-in-llms-omkn5wo4n) · Machine Learning Mastery · 2 upvotes · 0 comments --- Tags: [#deep-learning](https://daily.dev/tags/deep-learning), [#ai-inference](https://daily.dev/tags/ai-inference) [View this post on daily.dev](https://daily.dev/posts/why-kv-cache-stores-k-and-v-vectors-but-never-q--gz6gkpinu) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Why KV Cache Stores K and V Vectors But Never Q?","url":"https://daily.dev/posts/why-kv-cache-stores-k-and-v-vectors-but-never-q--gz6gkpinu","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/why-kv-cache-stores-k-and-v-vectors-but-never-q--gz6gkpinu"},"datePublished":"2026-08-30T20:21:25.268Z","dateModified":"2026-09-13T19:34:16.172Z","description":"A deep dive explains why the KV cache in LLM inference stores only key and value vectors, never query vectors. Walking through prefill and decoding steps token...","image":"https://substackcdn.com/image/fetch/$s_!Z-vk!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95bce90d-2cd2-41eb-b875-4d1b6fe4d11b_1211x1211.gif","thumbnailUrl":"https://substackcdn.com/image/fetch/$s_!Z-vk!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95bce90d-2cd2-41eb-b875-4d1b6fe4d11b_1211x1211.gif","isAccessibleForFree":true,"articleSection":"Daily Dose of Data Science | Avi Chawla | Substack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Daily Dose of Data Science | Avi Chawla | Substack","logo":"https://media.daily.dev/image/upload/s--4IHQgTOw--/f_auto/v1710503712/logos/dailydoseofds","url":"https://daily.dev/sources/dailydoseofds"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/why-kv-cache-stores-k-and-v-vectors-but-never-q--gz6gkpinu","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"deep-learning,ai-inference","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Daily Dose of Data Science | Avi Chawla | Substack","item":"https://daily.dev/sources/dailydoseofds"},{"@type":"ListItem","position":3,"name":"Why KV Cache Stores K and V Vectors But Never Q?"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/why-kv-cache-stores-k-and-v-vectors-but-never-q--gz6gkpinu#faq","mainEntity":[{"@type":"Question","name":"Why does the KV cache in transformer models store key and value vectors but not the query vector?","acceptedAnswer":{"@type":"Answer","text":"Each decode step only needs the query vector for the newest token position to compute that step's attention output, and it is never reused afterward, so caching it wastes memory. Keys and values, however, depend only on a token and the tokens before it under causal masking, so once computed they stay valid for every future decoding step and must be retained to avoid recomputation. daily.dev surfaces deep dives like this for engineers optimizing LLM inference and serving costs."}},{"@type":"Question","name":"What causes the first token in LLM generation to take longer than subsequent tokens?","acceptedAnswer":{"@type":"Answer","text":"The delay comes from prefill, the compute-heavy forward pass where all prompt tokens are processed in parallel with causal masking to produce query, key, and value vectors at every layer for every position. Because prefill computes attention across the entire prompt at once before any output token is generated, it dominates time-to-first-token (TTFT), while later tokens only require processing one new position. understanding prefill versus decode helps teams reasoning about LLM latency budgets on daily.dev."}},{"@type":"Question","name":"How fast is Datalab's Marker v2 PDF parsing pipeline and what model does it use?","acceptedAnswer":{"@type":"Answer","text":"Marker v2 uses a single 650-million-parameter model called Surya 2 that handles OCR, layout, reading order, and table recognition across more than 90 languages. On a single B200 GPU it sustains 2.9 pages per second in balanced mode, 7.4 in fast mode, and 23.7 with OCR disabled, by routing many lightweight CPU workers to one shared inference server instead of giving each worker its own model copy. developers comparing document-parsing pipelines can track benchmarks like this on daily.dev."}}]} ```