--- title: Prefix-Aware LLM Routing, clearly explained | daily.dev description: Prefix-aware routing can cut time to first token by about 170 times compared to random routing in multi-GPU LLM serving. In llm-d tests on 8 vLLM pods across... canonical: https://daily.dev/posts/prefix-aware-llm-routing-clearly-explained-bgzzvufwk twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Prefix-Aware LLM Routing, clearly explained | daily.dev og:description: Prefix-aware routing can cut time to first token by about 170 times compared to random routing in multi-GPU LLM serving. In llm-d tests on 8 vLLM pods across... og:url: https://daily.dev/posts/prefix-aware-llm-routing-clearly-explained-bgzzvufwk og:image: https://api.daily.dev/og/posts/BgZzvufwk.png og:image:alt: Prefix-Aware LLM Routing, clearly explained og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Prefix-Aware LLM Routing, clearly explained **[Daily Dose of Data Science \| Avi Chawla \| Substack](https://daily.dev/sources/dailydoseofds)** · 13 min read · 0 upvotes · 0 comments ## Summary Prefix-aware routing can cut time to first token by about 170 times compared to random routing in multi-GPU LLM serving. In llm-d tests on 8 vLLM pods across 16 H100s, random routing hit a p90 time to first token of 92.6 seconds, while precise prefix-aware routing hit 0.54 seconds, with throughput roughly doubling from 4,429 to 8,730 tokens per second. The router hashes prompts block by block (vLLM uses 16-token blocks) and maintains a tiny block-to-worker index, about 339 KB for a 365 GB KV cache pool. Load-aware routing alone performed no better than random, since it ignores cache locality. NVIDIA Dynamo's KV router balances cache hits against worker load using a cost formula combining uncached prefill blocks and decode load. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much faster is prefix-aware routing compared to random routing for LLM inference across multiple GPUs? Precise prefix-aware routing cut p90 time to first token to 0.54 seconds versus 92.6 seconds for random routing, about 170 times faster, in tests on 8 vLLM pods across 16 H100 GPUs. Output throughput also roughly doubled, from 4,429 to 8,730 tokens per second. Load-aware routing alone performed no better than random routing because it ignores cache locality entirely. _Evaluating LLM serving infrastructure choices gets easier with benchmark details like these tracked on daily.dev._ ### How does vLLM hash KV cache blocks for prefix matching? vLLM stores the KV cache in fixed-size blocks of 16 tokens by default and only caches complete blocks. Each block's hash is built from its parent block's hash plus its own token IDs, so matching a later block's hash confirms every block before it also matched, turning a block lookup into a prefix lookup without inspecting the actual KV tensors. _Developers tuning vLLM serving performance can keep up with these caching internals through daily.dev._ ### How does NVIDIA Dynamo's KV router decide which worker gets a request when cache hits and load conflict? Dynamo's KV router computes a cost per worker as uncached prefill blocks plus existing decode load, then picks the lowest total. In its documented example with 10 prefill blocks, Worker 3 has the best cache match (8 cached blocks) but Worker 2 wins with a lower combined cost of 10, since Dynamo treats more cache credit as lowering time to first token and less credit as spreading load more evenly. _Teams comparing LLM routing strategies can track these tradeoffs through daily.dev before picking one._ ## Similar posts on daily.dev - [The Inference Tax: How Prefix-Aware Routing Eliminates the Hidden Cost of LLMs at Scale](https://daily.dev/posts/the-inference-tax-how-prefix-aware-routing-eliminates-the-hidden-cost-of-llms-at-scale-eej7jbkjc) · DigitalOcean · 0 upvotes · 0 comments - [GKE Inference Gateway prefix caching accelerates AI inference](https://daily.dev/posts/gke-inference-gateway-prefix-caching-accelerates-ai-inference-qyjrx3igz) · Google Cloud · 1 upvotes · 0 comments - [Intelligent inference scheduling with llm-d on Red Hat AI](https://daily.dev/posts/intelligent-inference-scheduling-with-llm-d-on-red-hat-ai-q8lqp3mss) · Red Hat Developer · 0 upvotes · 0 comments - [Multi-Provider LLM Routing Is Not a Problem, It's Your Architecture: Inference in Production Series](https://daily.dev/posts/multi-provider-llm-routing-is-not-a-problem-it-s-your-architecture-inference-in-production-series-b8ljpdbw8) · DigitalOcean Community · 0 upvotes · 0 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/prefix-aware-llm-routing-clearly-explained-bgzzvufwk) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Prefix-Aware LLM Routing, clearly explained","url":"https://daily.dev/posts/prefix-aware-llm-routing-clearly-explained-bgzzvufwk","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/prefix-aware-llm-routing-clearly-explained-bgzzvufwk"},"datePublished":"2026-10-29T20:47:16.312Z","dateModified":"2026-10-29T20:48:08.785Z","description":"Prefix-aware routing can cut time to first token by about 170 times compared to random routing in multi-GPU LLM serving. In llm-d tests on 8 vLLM pods across...","image":"https://media.daily.dev/image/fetch/s--xaw6RJyT--/f_auto,pg_1,q_auto/https://substackcdn.com/image/fetch/%24s_%219iec%21%2Cw_1200%2Ch_675%2Cc_fill%2Cf_jpg%2Cq_auto:good%2Cfl_progressive:steep%2Cg_auto/https%253A%252F%252Fsubstack-post-media.s3.amazonaws.com%252Fpublic%252Fimages%252F4d894230-732b-485d-9d5e-3702526b42eb_1100x928.gif?_a=BAMAMiiw0","thumbnailUrl":"https://media.daily.dev/image/fetch/s--xaw6RJyT--/f_auto,pg_1,q_auto/https://substackcdn.com/image/fetch/%24s_%219iec%21%2Cw_1200%2Ch_675%2Cc_fill%2Cf_jpg%2Cq_auto:good%2Cfl_progressive:steep%2Cg_auto/https%253A%252F%252Fsubstack-post-media.s3.amazonaws.com%252Fpublic%252Fimages%252F4d894230-732b-485d-9d5e-3702526b42eb_1100x928.gif?_a=BAMAMiiw0","isAccessibleForFree":true,"articleSection":"Daily Dose of Data Science | Avi Chawla | Substack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Daily Dose of Data Science | Avi Chawla | Substack","logo":"https://media.daily.dev/image/upload/s--4IHQgTOw--/f_auto/v1710503712/logos/dailydoseofds","url":"https://daily.dev/sources/dailydoseofds"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/prefix-aware-llm-routing-clearly-explained-bgzzvufwk","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"aws,ai-inference,vllm","timeRequired":"PT13M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Daily Dose of Data Science | Avi Chawla | Substack","item":"https://daily.dev/sources/dailydoseofds"},{"@type":"ListItem","position":3,"name":"Prefix-Aware LLM Routing, clearly explained"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/prefix-aware-llm-routing-clearly-explained-bgzzvufwk#faq","mainEntity":[{"@type":"Question","name":"How much faster is prefix-aware routing compared to random routing for LLM inference across multiple GPUs?","acceptedAnswer":{"@type":"Answer","text":"Precise prefix-aware routing cut p90 time to first token to 0.54 seconds versus 92.6 seconds for random routing, about 170 times faster, in tests on 8 vLLM pods across 16 H100 GPUs. Output throughput also roughly doubled, from 4,429 to 8,730 tokens per second. Load-aware routing alone performed no better than random routing because it ignores cache locality entirely. Evaluating LLM serving infrastructure choices gets easier with benchmark details like these tracked on daily.dev."}},{"@type":"Question","name":"How does vLLM hash KV cache blocks for prefix matching?","acceptedAnswer":{"@type":"Answer","text":"vLLM stores the KV cache in fixed-size blocks of 16 tokens by default and only caches complete blocks. Each block's hash is built from its parent block's hash plus its own token IDs, so matching a later block's hash confirms every block before it also matched, turning a block lookup into a prefix lookup without inspecting the actual KV tensors. Developers tuning vLLM serving performance can keep up with these caching internals through daily.dev."}},{"@type":"Question","name":"How does NVIDIA Dynamo's KV router decide which worker gets a request when cache hits and load conflict?","acceptedAnswer":{"@type":"Answer","text":"Dynamo's KV router computes a cost per worker as uncached prefill blocks plus existing decode load, then picks the lowest total. In its documented example with 10 prefill blocks, Worker 3 has the best cache match (8 cached blocks) but Worker 2 wins with a lower combined cost of 10, since Dynamo treats more cache credit as lowering time to first token and less credit as spreading load more evenly. Teams comparing LLM routing strategies can track these tradeoffs through daily.dev before picking one."}}]} ```