--- title: How a GPU Actually Works | daily.dev description: A memory-bound explanation of why GPUs rated for near a thousand trillion operations per second still produce only a few dozen tokens per second when serving a... canonical: https://daily.dev/posts/how-a-gpu-actually-works-t655zbnjv twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How a GPU Actually Works | daily.dev og:description: A memory-bound explanation of why GPUs rated for near a thousand trillion operations per second still produce only a few dozen tokens per second when serving a... og:url: https://daily.dev/posts/how-a-gpu-actually-works-t655zbnjv og:image: https://api.daily.dev/og/posts/t655zbnjV.png og:image:alt: How a GPU Actually Works og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How a GPU Actually Works **[Daily Dose of Data Science \| Avi Chawla \| Substack](https://daily.dev/sources/dailydoseofds)** · 6 min read · 0 upvotes · 0 comments ## Summary A memory-bound explanation of why GPUs rated for near a thousand trillion operations per second still produce only a few dozen tokens per second when serving a 70B model, tracing the bottleneck to the ratio of arithmetic performed per byte fetched. Also covers a newsletter roundup: an open-source mock server (aimock) that keeps CI tests from silently drifting from real LLM API schemas, and CacheBlend, a research technique from LMCache that fixes prefix caching's failure to reuse cached documents across reordering or multi-document RAG queries, claiming 2-4x speedups with no quality loss. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does a GPU rated for trillions of operations per second only generate a few dozen tokens per second when serving a 70B parameter model? Token generation is memory-bound, not compute-bound. Current hardware breaks even near 300 operations per byte fetched at 16-bit precision, but generating a token performs only about two operations per weight read, landing near a ratio of one, roughly 300 times below the break-even point. For a 70B model with 140GB of weights over 3.3TB/s of bandwidth, that works out to 42 milliseconds per token, or about 24 tokens per second, regardless of how fast the arithmetic units are. _Anyone tuning LLM inference throughput can find deeper breakdowns like this GPU memory bottleneck explainer on daily.dev._ ### Why does prefix caching fail when RAG documents appear in different orders or when combining previously separate documents? Prefix caching requires the cached region to be an exact byte-for-byte match of the new request's prefix, so any reordering or new combination of documents causes a full cache miss and recomputation from scratch. Alibaba Cloud production data shows only 10% of KV cache blocks serve 77% of all cache hits, with the rest sitting unused because prefix matching won't allow reuse in a different context. _Teams optimizing RAG pipeline costs can track caching techniques like this on daily.dev._ ### What is CacheBlend and how much faster is it than standard prefix caching for multi-document queries? CacheBlend is a caching technique from LMCache, winner of the EuroSys 2025 Best Paper Award, that reuses every document's KV cache as-is and selectively recomputes only the small fraction of boundary tokens that carry cross-document attention connections, instead of recomputing everything after the first cached document. This delivers 2 to 4x faster processing on multi-document queries with no quality loss, and it remains cached regardless of document order, unlike standard prefix caching. _Developers evaluating KV-cache strategies for RAG systems can follow tools like LMCache on daily.dev._ ## Similar posts on daily.dev - [Your GPU Is a Kitchen With One Tiny Door](https://daily.dev/posts/your-gpu-is-a-kitchen-with-one-tiny-door-rbbxjvpyf) · Medium · 1 upvotes · 0 comments - [Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't \(and Where They Slow Down\)](https://daily.dev/posts/unified-memory-explained-why-mini-pcs-can-run-70b-models-a-big-gpu-can-t-and-where-they-slow-down-rinvfgbab) · Hacker News · 1 upvotes · 1 comments - [5 Techniques to Optimize LLMs in Production](https://daily.dev/posts/5-techniques-to-optimize-llms-in-production-lmz4pypvm) · Daily Dose of Data Science \| Avi Chawla \| Substack · 0 upvotes · 0 comments - [The Strangest Bottleneck in Modern LLMs](https://daily.dev/posts/the-strangest-bottleneck-in-modern-llms-wg80htes6) · Towards Data Science · 1 upvotes · 0 comments --- Tags: [#data-science](https://daily.dev/tags/data-science), [#gpu](https://daily.dev/tags/gpu), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/how-a-gpu-actually-works-t655zbnjv) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How a GPU Actually Works","url":"https://daily.dev/posts/how-a-gpu-actually-works-t655zbnjv","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-a-gpu-actually-works-t655zbnjv"},"datePublished":"2026-08-17T21:02:36.435Z","dateModified":"2026-09-13T21:34:32.716Z","description":"A memory-bound explanation of why GPUs rated for near a thousand trillion operations per second still produce only a few dozen tokens per second when serving a...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/40c353883d79b5c0262e69aab2e984c5?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/40c353883d79b5c0262e69aab2e984c5?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Daily Dose of Data Science | Avi Chawla | Substack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Daily Dose of Data Science | Avi Chawla | Substack","logo":"https://media.daily.dev/image/upload/s--4IHQgTOw--/f_auto/v1710503712/logos/dailydoseofds","url":"https://daily.dev/sources/dailydoseofds"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-a-gpu-actually-works-t655zbnjv","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-science,gpu,ai-inference,vllm","timeRequired":"PT6M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Daily Dose of Data Science | Avi Chawla | Substack","item":"https://daily.dev/sources/dailydoseofds"},{"@type":"ListItem","position":3,"name":"How a GPU Actually Works"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-a-gpu-actually-works-t655zbnjv#faq","mainEntity":[{"@type":"Question","name":"Why does a GPU rated for trillions of operations per second only generate a few dozen tokens per second when serving a 70B parameter model?","acceptedAnswer":{"@type":"Answer","text":"Token generation is memory-bound, not compute-bound. Current hardware breaks even near 300 operations per byte fetched at 16-bit precision, but generating a token performs only about two operations per weight read, landing near a ratio of one, roughly 300 times below the break-even point. For a 70B model with 140GB of weights over 3.3TB/s of bandwidth, that works out to 42 milliseconds per token, or about 24 tokens per second, regardless of how fast the arithmetic units are. Anyone tuning LLM inference throughput can find deeper breakdowns like this GPU memory bottleneck explainer on daily.dev."}},{"@type":"Question","name":"Why does prefix caching fail when RAG documents appear in different orders or when combining previously separate documents?","acceptedAnswer":{"@type":"Answer","text":"Prefix caching requires the cached region to be an exact byte-for-byte match of the new request's prefix, so any reordering or new combination of documents causes a full cache miss and recomputation from scratch. Alibaba Cloud production data shows only 10% of KV cache blocks serve 77% of all cache hits, with the rest sitting unused because prefix matching won't allow reuse in a different context. Teams optimizing RAG pipeline costs can track caching techniques like this on daily.dev."}},{"@type":"Question","name":"What is CacheBlend and how much faster is it than standard prefix caching for multi-document queries?","acceptedAnswer":{"@type":"Answer","text":"CacheBlend is a caching technique from LMCache, winner of the EuroSys 2025 Best Paper Award, that reuses every document's KV cache as-is and selectively recomputes only the small fraction of boundary tokens that carry cross-document attention connections, instead of recomputing everything after the first cached document. This delivers 2 to 4x faster processing on multi-document queries with no quality loss, and it remains cached regardless of document order, unlike standard prefix caching. Developers evaluating KV-cache strategies for RAG systems can follow tools like LMCache on daily.dev."}}]} ```