A detailed technical walkthrough of speculative decoding in vLLM running on AMD Instinct MI300X and MI355X GPUs with ROCm. It covers five drafting approaches (native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark), explains how each draft component uses target-model information and generates candidate tokens, and shows how to configure them via --speculative-config. Extensive benchmarks across Gemma, Qwen, MiniMax, and Kimi models show throughput gains ranging from below baseline to over 2.8x depending on model, workload, and proposal length (num_speculative_tokens). Practical tuning guidance covers sweeping proposal lengths, monitoring acceptance rate and mean accepted length, and matching sweep depth to workload type. A section also outlines how to train a new speculator using the vLLM Speculators toolkit.
Table of contents
IntroductionThe autoregressive decoding baselineCore idea of speculative decodingHow the drafting methods workHow to enable speculative decoding in vLLMWhere to find the pretrained draft modelsExperimental setup and measurementsTuning considerationsTraining a speculator for a new target modelSummaryFuture workReferencesAppendixAcknowledgementsDisclaimerQuestions this post answers
What speculative decoding methods does vLLM support for AMD GPUs?
vLLM's --speculative-config supports four method values: mtp, eagle3, dflash, and dspark, covering native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark drafting approaches. Native MTP requires no separate draft checkpoint, while the others need a matching speculator model specified via the model field, plus a chosen num_speculative_tokens value. Engineers tuning inference throughput can follow ongoing vLLM tooling coverage on daily.dev.
How does DFlash differ from EAGLE-3 in speculative decoding?
DFlash predicts an entire block of future token positions in parallel in a single forward pass, while EAGLE-3 generates draft tokens autoregressively, with each token depending on the previous one. Both fuse hidden states from multiple target-model layers, but DFlash injects this context as Key/Value pairs available in every draft layer, whereas EAGLE-3 combines it only at the input with the sampled-token embedding. Teams choosing between parallel and sequential drafting strategies can track comparisons like this on daily.dev.
What throughput speedups did DFlash and Gemma 4 MTP achieve on gemma-4-26B-A4B-it?
On gemma-4-26B-A4B-it, DFlash reached up to 2.87x throughput on MATH500 and 2.79x on HumanEval, while Gemma 4 MTP reached 2.74x on GSM8K and 2.62x on MBPP. EAGLE-3 measurements on the same model ranged from 2.11x to 2.27x across the four datasets, with proposal length (N) tuned per workload during testing on AMD MI300X/MI355X GPUs. Practitioners benchmarking inference speedups can follow real-world throughput numbers on daily.dev.