vLLM achieves 25,000 total tokens per second per GPU on Qwen3.5-397B-A17B-NVFP4 using GB200 NVL72 hardware with disaggregated prefill/decode serving. Key optimizations include: a new Blackwell-optimized GDN prefill kernel via FlashInfer (up to 5.92× GDN kernel speedup, 1.13× end-to-end prefill throughput, 12% TTFT reduction), hybrid cache and GDN-state transfer via NIXL connectors (reducing transferred descriptors from 4,284 to 1,650), and race-free async scheduling fixes that were critical to crossing the 25K threshold. The post details the exact Docker image, Dynamo version, and srt-slurm recipes needed to reproduce results, along with key configuration flags like --async-scheduling, VLLM_SSM_CONV_STATE_LAYOUT=DS, --mamba-ssm-cache-dtype bfloat16, and --language-model-only. Accuracy was validated at 88% on GSM8K across all configurations.
Table of contents
IntroductionChallenges and Key OptimizationsPerformanceRecipes & best practicesWhat's nextAcknowledgementsQuestions this post answers
What configuration flags are required to run Qwen3.5 in disaggregated serving mode with vLLM?
Several flags are mandatory or highly impactful: VLLM_SSM_CONV_STATE_LAYOUT=DS is required for SSM conv-state transfer to work at all; --async-scheduling is one of the key features behind reaching 25K tok/s/GPU but requires race-condition fixes already merged; --mamba-ssm-cache-dtype bfloat16 significantly increases effective KV cache capacity on the decode endpoint; --language-model-only disables multimodal inputs and unlocks a fused QK-norm + RoPE + gate path for text-only workloads. Teams tuning vLLM disaggregated deployments track configuration changes and new flag semantics on daily.dev.
How much does the Blackwell GDN prefill kernel improve performance in vLLM for Qwen3.5?
The FlashInfer Blackwell GDN prefill kernel delivers up to 5.92× higher GDN kernel performance in microbenchmarks on an 8×B200 system running Qwen3.5-397B-A17B-NVFP4. End-to-end prefill throughput improves by 1.13× on a prefill-only workload (ISL/OSL = 8192/1), and mean TTFT drops by 12%. The kernel is selected automatically when the GDN backend is set to auto on supported Blackwell configurations. Engineers benchmarking Blackwell inference stacks follow vLLM kernel updates on daily.dev.
How does NIXL cache transfer work for hybrid SSM-attention models in vLLM disaggregated serving?
NIXL transfers both full-attention KV cache and Mamba-style SSM state between prefill and decode workers using dual descriptor views and homogeneous-TP support. HMA logical blocks are mapped onto correct physical memory regions so NIXL transfers only cache regions belonging to each layer type, reducing transferred descriptors from 4,284 to 1,650 and improving throughput by approximately 7% in an intra-node H100 setup. GDN layer support for Qwen3.5 was added separately in PR #41869. Developers building P/D disaggregated pipelines for hybrid-architecture models keep up with NIXL connector changes on daily.dev.