vLLM adds adaptive verification for DSpark speculative decoding, using a per-token confidence head to dynamically size the draft-verification budget instead of a static num_speculative_tokens value. Since per-position acceptance probability decays sharply within a drafted block, verifying every token wastes compute at high concurrency; the new scheduler computes a global top-B selection across requests each step to maximize expected tokens per unit step time, using a profiled cost table and CUDA-graph-aware budget selection. Benchmarks on DeepSeek-V4-Pro-0813 (TP=8, 8×B300) show adaptive verification stays on the Pareto frontier from concurrency 1 to 256, matching long fixed-block behavior at low concurrency and short-block behavior at high concurrency without manual tuning. The feature landed in vLLM PR #47808 as enable_adaptive_verification, with current limitations around SM100-only support, no eager/LoRA/pipeline-parallel support, and no output logprobs.
Table of contents
The problemScheduling the budgetVarlen decode CUDA graphsThe cost modelResultsLimitationsAppendix: reproducingAcknowledgmentsQuestions this post answers
How does vLLM decide how many speculative decoding tokens to verify per step with DSpark?
vLLM's adaptive verification feature uses DSpark's confidence head to score each drafted token's survival probability, then picks a global top-B set of draft tokens across all requests to verify by maximizing expected tokens per unit of step time. This replaces a fixed num_speculative_tokens value, since acceptance rates and the optimal budget shift with concurrency and workload. It shipped in PR #47808 as the enable_adaptive_verification flag. daily.dev surfaces engineering deep dives like this for teams tuning LLM inference throughput.
Why does speculative decoding throughput drop at high batch sizes in LLM inference servers?
At large batch sizes, draft tokens compete with real tokens for the same GPU compute, so rejected draft tokens waste useful capacity rather than being nearly free as they are at batch size 1. On DeepSeek-V4-Pro-0813, the last token of a 7-token speculative block survives verification less than 10% of the time versus over 70% for the first token, making a fixed speculation length wasteful once the GPU becomes compute-bound. Engineers weighing speculative decoding trade-offs can track these throughput findings on daily.dev.
What GPU setup and results were used to benchmark vLLM's adaptive verification feature?
Benchmarks ran DeepSeek-V4-Pro-0813 with tensor parallel size 8 on 8×B300 GPUs (SM100), expert parallel, FP8 KV cache, max_model_len 16384, and max_cudagraph_capture_size 4096 on vLLM main at commit 73b8394. Across 880 prompts swept from concurrency 1 to 256, adaptive verification stayed on the Pareto frontier throughout, outperforming both fixed-length speculation and no speculation at the extremes. daily.dev helps infra engineers keep up with benchmark data behind inference serving decisions.