A memory-bound explanation of why GPUs rated for near a thousand trillion operations per second still produce only a few dozen tokens per second when serving a 70B model, tracing the bottleneck to the ratio of arithmetic performed per byte fetched. Also covers a newsletter roundup: an open-source mock server (aimock) that keeps CI tests from silently drifting from real LLM API schemas, and CacheBlend, a research technique from LMCache that fixes prefix caching's failure to reuse cached documents across reordering or multi-document RAG queries, claiming 2-4x speedups with no quality loss.
Table of contents
Mock infrastructure for AI apps (open-source)How a GPU actually works Prefix caching vs CacheBlend Questions this post answers
Why does a GPU rated for near a thousand trillion operations per second only generate a few dozen tokens per second on a 70B model?
Token generation is memory-bandwidth bound, not compute bound. Current 16-bit hardware breaks even near 300 operations per byte fetched, but generating a token performs only about two operations per weight read, landing near a ratio of one, roughly 300 times below break-even. For a 70B model with 140GB of weights and 3.3TB/s bandwidth, that works out to about 42 milliseconds per token, or roughly 24 tokens per second, while arithmetic units sit idle waiting for data. daily.dev surfaces deep dives like this for engineers optimizing LLM inference throughput.
What is the difference between standard prefix caching and CacheBlend for LLM KV caches?
Standard prefix caching requires the cached portion to be an exact byte-for-byte prefix of the new request, so reordering documents or combining separately cached documents causes a full cache miss and recomputation. CacheBlend, developed by LMCache and awarded EuroSys 2025 Best Paper, instead reuses every document's cache as-is and selectively recomputes only the small fraction of boundary tokens that carry cross-document connections, giving 2 to 4x faster processing on multi-document queries with no quality loss. daily.dev helps teams comparing caching strategies keep up with new LLM inference research.
How can I prevent CI tests against mocked LLM API responses from drifting out of sync with the real provider schema?
Use a mock server whose fake responses are validated daily against the real API's actual output and the official client library's type definitions, rather than a static saved dummy response. The open-source aimock project does this: a single repo runs the real API calls on its own keys, and when a schema mismatch is found, a patch version ships to npm so every project using it gets the corrected schema on upgrade. daily.dev keeps developers building AI app pipelines aware of tools like this before CI drifts silently.