--- title: "How a GPU Actually Works" url: https://daily.dev/posts/how-a-gpu-actually-works-t655zbnjv source_url: https://blog.dailydoseofds.com/p/how-a-gpu-actually-works type: article source: "Daily Dose of Data Science | Avi Chawla | Substack" published: 2026-08-17T21:02:36.435Z updated: 2026-08-17T21:03:03.892Z tags: ["data-science", "gpu", "ai-inference", "vllm"] reading_time: 6 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How a GPU Actually Works **[Daily Dose of Data Science \| Avi Chawla \| Substack](https://daily.dev/sources/dailydoseofds)** · 6 min read · 0 upvotes · 0 comments ## Summary A memory-bound explanation of why GPUs rated for near a thousand trillion operations per second still produce only a few dozen tokens per second when serving a 70B model, tracing the bottleneck to the ratio of arithmetic performed per byte fetched. Also covers a newsletter roundup: an open-source mock server (aimock) that keeps CI tests from silently drifting from real LLM API schemas, and CacheBlend, a research technique from LMCache that fixes prefix caching's failure to reuse cached documents across reordering or multi-document RAG queries, claiming 2-4x speedups with no quality loss. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does a GPU rated for near a thousand trillion operations per second only generate a few dozen tokens per second on a 70B model? Token generation is memory-bandwidth bound, not compute bound. Current 16-bit hardware breaks even near 300 operations per byte fetched, but generating a token performs only about two operations per weight read, landing near a ratio of one, roughly 300 times below break-even. For a 70B model with 140GB of weights and 3.3TB/s bandwidth, that works out to about 42 milliseconds per token, or roughly 24 tokens per second, while arithmetic units sit idle waiting for data. _daily.dev surfaces deep dives like this for engineers optimizing LLM inference throughput._ ### What is the difference between standard prefix caching and CacheBlend for LLM KV caches? Standard prefix caching requires the cached portion to be an exact byte-for-byte prefix of the new request, so reordering documents or combining separately cached documents causes a full cache miss and recomputation. CacheBlend, developed by LMCache and awarded EuroSys 2025 Best Paper, instead reuses every document's cache as-is and selectively recomputes only the small fraction of boundary tokens that carry cross-document connections, giving 2 to 4x faster processing on multi-document queries with no quality loss. _daily.dev helps teams comparing caching strategies keep up with new LLM inference research._ ### How can I prevent CI tests against mocked LLM API responses from drifting out of sync with the real provider schema? Use a mock server whose fake responses are validated daily against the real API's actual output and the official client library's type definitions, rather than a static saved dummy response. The open-source aimock project does this: a single repo runs the real API calls on its own keys, and when a schema mismatch is found, a patch version ships to npm so every project using it gets the corrected schema on upgrade. _daily.dev keeps developers building AI app pipelines aware of tools like this before CI drifts silently._ ## Similar posts on daily.dev - [Your GPU Is a Kitchen With One Tiny Door](https://daily.dev/posts/your-gpu-is-a-kitchen-with-one-tiny-door-rbbxjvpyf) · Medium · 1 upvotes · 0 comments - [Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't \(and Where They Slow Down\)](https://daily.dev/posts/unified-memory-explained-why-mini-pcs-can-run-70b-models-a-big-gpu-can-t-and-where-they-slow-down-rinvfgbab) · Hacker News · 1 upvotes · 1 comments --- Tags: [#data-science](https://daily.dev/tags/data-science), [#gpu](https://daily.dev/tags/gpu), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/how-a-gpu-actually-works-t655zbnjv)