--- title: vLLM Reaches 25K Total TPS/GPU on Qwen3.5 | daily.dev description: vLLM achieves 25,000 total tokens per second per GPU on Qwen3.5-397B-A17B-NVFP4 using GB200 NVL72 hardware with disaggregated prefill/decode serving. Key... canonical: https://daily.dev/posts/vllm-reaches-25k-total-tps-gpu-on-qwen3-5-hjwogrfb1 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: vLLM Reaches 25K Total TPS/GPU on Qwen3.5 | daily.dev og:description: vLLM achieves 25,000 total tokens per second per GPU on Qwen3.5-397B-A17B-NVFP4 using GB200 NVL72 hardware with disaggregated prefill/decode serving. Key... og:url: https://daily.dev/posts/vllm-reaches-25k-total-tps-gpu-on-qwen3-5-hjwogrfb1 og:image: https://api.daily.dev/og/posts/hJWoGrFB1.png og:image:alt: vLLM Reaches 25K Total TPS/GPU on Qwen3.5 og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # vLLM Reaches 25K Total TPS/GPU on Qwen3.5 **[vLLM](https://daily.dev/sources/vllm)** · 9 min read · 1 upvotes · 0 comments ## Summary vLLM achieves 25,000 total tokens per second per GPU on Qwen3.5-397B-A17B-NVFP4 using GB200 NVL72 hardware with disaggregated prefill/decode serving. Key optimizations include: a new Blackwell-optimized GDN prefill kernel via FlashInfer (up to 5.92× GDN kernel speedup, 1.13× end-to-end prefill throughput, 12% TTFT reduction), hybrid cache and GDN-state transfer via NIXL connectors (reducing transferred descriptors from 4,284 to 1,650), and race-free async scheduling fixes that were critical to crossing the 25K threshold. The post details the exact Docker image, Dynamo version, and srt-slurm recipes needed to reproduce results, along with key configuration flags like --async-scheduling, VLLM_SSM_CONV_STATE_LAYOUT=DS, --mamba-ssm-cache-dtype bfloat16, and --language-model-only. Accuracy was validated at 88% on GSM8K across all configurations. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What vLLM flag enables the Blackwell-optimized GDN prefill kernel for Qwen3.5? Setting the GDN prefill backend to flashinfer via --gdn-prefill-backend flashinfer enables the FlashInfer Blackwell GDN prefill kernel. On supported Blackwell configurations vLLM selects this path automatically when the backend is set to auto. On an 8xB200 system running Qwen3.5-397B-A17B-NVFP4, this delivered up to 5.92x higher GDN kernel performance, 1.13x higher end-to-end prefill throughput, and a 12% reduction in mean TTFT. _Engineers tuning Qwen3.5 inference performance can track vLLM kernel and flag changes like this through daily.dev._ ### Why does async scheduling need special race-condition fixes in vLLM disaggregated serving? Async scheduling caused KV block transfer race conditions in disaggregated prefill/decode serving that made accuracy collapse to zero for hybrid attention models. Two patches fixed this: one addressing the PD async scheduling race for hybrid attention models, and another deferring block freeing until in-flight steps finish under async scheduling with a PD KV consumer. Async scheduling turned out to be a key feature behind reaching 25K tokens/sec per GPU once fixed. _Teams debugging accuracy regressions in async disaggregated serving can follow fixes like these on daily.dev._ ### What environment variable is required for SSM conv-state transfer in vLLM disaggregated serving? VLLM_SSM_CONV_STATE_LAYOUT=DS is mandatory for SSM models such as Qwen3.5 in disaggregated serving, since conv-state transfer does not function without it. Additionally, --mamba-ssm-cache-dtype bfloat16 significantly increases effective KV cache capacity on the decode endpoint, and prefix caching should be disabled since it provides no benefit on random datasets used for benchmarking. _Anyone configuring hybrid SSM model serving can keep up with vLLM recipe changes via daily.dev._ ## Similar posts on daily.dev - [Implementing vLLM in Qwen Serving](https://daily.dev/posts/implementing-vllm-in-qwen-serving-q8njflolx) · Medium · 0 upvotes · 0 comments - [How I massively improved my AI inference performance without buying new hardware](https://daily.dev/posts/how-i-massively-improved-my-ai-inference-performance-without-buying-new-hardware-jgyj1odfm) · Red Hat Developer · 0 upvotes · 0 comments --- Tags: [#ai-inference](https://daily.dev/tags/ai-inference), [#qwen](https://daily.dev/tags/qwen), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/vllm-reaches-25k-total-tps-gpu-on-qwen3-5-hjwogrfb1) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"vLLM Reaches 25K Total TPS/GPU on Qwen3.5","url":"https://daily.dev/posts/vllm-reaches-25k-total-tps-gpu-on-qwen3-5-hjwogrfb1","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/vllm-reaches-25k-total-tps-gpu-on-qwen3-5-hjwogrfb1"},"datePublished":"2026-08-07T12:26:00.959Z","dateModified":"2026-09-13T21:21:44.481Z","description":"vLLM achieves 25,000 total tokens per second per GPU on Qwen3.5-397B-A17B-NVFP4 using GB200 NVL72 hardware with disaggregated prefill/decode serving. Key...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c8a0a7758586ff0f2af3641536da588e?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c8a0a7758586ff0f2af3641536da588e?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"vLLM","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"vLLM","logo":"https://media.daily.dev/image/upload/s--hTxEuls9--/f_auto/v1744613054/logos/vllm","url":"https://daily.dev/sources/vllm"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/vllm-reaches-25k-total-tps-gpu-on-qwen3-5-hjwogrfb1","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-inference,qwen,vllm","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"vLLM","item":"https://daily.dev/sources/vllm"},{"@type":"ListItem","position":3,"name":"vLLM Reaches 25K Total TPS/GPU on Qwen3.5"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/vllm-reaches-25k-total-tps-gpu-on-qwen3-5-hjwogrfb1#faq","mainEntity":[{"@type":"Question","name":"What vLLM flag enables the Blackwell-optimized GDN prefill kernel for Qwen3.5?","acceptedAnswer":{"@type":"Answer","text":"Setting the GDN prefill backend to flashinfer via --gdn-prefill-backend flashinfer enables the FlashInfer Blackwell GDN prefill kernel. On supported Blackwell configurations vLLM selects this path automatically when the backend is set to auto. On an 8xB200 system running Qwen3.5-397B-A17B-NVFP4, this delivered up to 5.92x higher GDN kernel performance, 1.13x higher end-to-end prefill throughput, and a 12% reduction in mean TTFT. Engineers tuning Qwen3.5 inference performance can track vLLM kernel and flag changes like this through daily.dev."}},{"@type":"Question","name":"Why does async scheduling need special race-condition fixes in vLLM disaggregated serving?","acceptedAnswer":{"@type":"Answer","text":"Async scheduling caused KV block transfer race conditions in disaggregated prefill/decode serving that made accuracy collapse to zero for hybrid attention models. Two patches fixed this: one addressing the PD async scheduling race for hybrid attention models, and another deferring block freeing until in-flight steps finish under async scheduling with a PD KV consumer. Async scheduling turned out to be a key feature behind reaching 25K tokens/sec per GPU once fixed. Teams debugging accuracy regressions in async disaggregated serving can follow fixes like these on daily.dev."}},{"@type":"Question","name":"What environment variable is required for SSM conv-state transfer in vLLM disaggregated serving?","acceptedAnswer":{"@type":"Answer","text":"VLLM_SSM_CONV_STATE_LAYOUT=DS is mandatory for SSM models such as Qwen3.5 in disaggregated serving, since conv-state transfer does not function without it. Additionally, --mamba-ssm-cache-dtype bfloat16 significantly increases effective KV cache capacity on the decode endpoint, and prefix caching should be disabled since it provides no benefit on random datasets used for benchmarking. Anyone configuring hybrid SSM model serving can keep up with vLLM recipe changes via daily.dev."}}]} ```