--- title: Exploring Speculative Decoding in vLLM on AMD GPUs description: A detailed technical walkthrough of speculative decoding in vLLM running on AMD Instinct MI300X and MI355X GPUs with ROCm. It covers five drafting approaches... canonical: https://daily.dev/posts/exploring-speculative-decoding-in-vllm-on-amd-gpus-3xgpvjp6n twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Exploring Speculative Decoding in vLLM on AMD GPUs | daily.dev og:description: A detailed technical walkthrough of speculative decoding in vLLM running on AMD Instinct MI300X and MI355X GPUs with ROCm. It covers five drafting approaches... og:url: https://daily.dev/posts/exploring-speculative-decoding-in-vllm-on-amd-gpus-3xgpvjp6n og:image: https://api.daily.dev/og/posts/3xGpvJP6n.png og:image:alt: Exploring Speculative Decoding in vLLM on AMD GPUs og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Exploring Speculative Decoding in vLLM on AMD GPUs **[vLLM](https://daily.dev/sources/vllm)** · 67 min read · 1 upvotes · 0 comments ## Summary A detailed technical walkthrough of speculative decoding in vLLM running on AMD Instinct MI300X and MI355X GPUs with ROCm. It covers five drafting approaches (native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark), explains how each draft component uses target-model information and generates candidate tokens, and shows how to configure them via --speculative-config. Extensive benchmarks across Gemma, Qwen, MiniMax, and Kimi models show throughput gains ranging from below baseline to over 2.8x depending on model, workload, and proposal length (num_speculative_tokens). Practical tuning guidance covers sweeping proposal lengths, monitoring acceptance rate and mean accepted length, and matching sweep depth to workload type. A section also outlines how to train a new speculator using the vLLM Speculators toolkit. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I configure speculative decoding in vLLM with DFlash? Set the speculative-config method to dflash and point the model field to a matching DFlash speculator checkpoint, along with num_speculative_tokens for the proposal length. DFlash predicts a whole block of masked positions in parallel using an anchor token, converting fused target-model hidden states into extra key and value information available in every draft-network layer. Many DFlash checkpoints are trained with a fixed block size, and N=7 was frequently among the higher-throughput settings across tested models. _Anyone tuning speculative decoding configs can track method-specific setup details like these on daily.dev._ ### What throughput speedup does EAGLE-3 speculative decoding achieve versus DFlash on the same model? On gemma-4-26B-A4B-it, EAGLE-3 measurements ranged from 2.11x to 2.27x across four datasets, while DFlash reached higher peaks of 2.87x on MATH500 and 2.79x on HumanEval. On Kimi-K2.5, EAGLE-3 reached up to 2.33x while DFlash reached up to 2.68x. DFlash's advantage generally grows with longer proposal lengths (commonly N=7), whereas EAGLE-3's sequential drafting favors shorter proposals like N=4. _Comparing speculative decoding methods before picking one is easier when tracking benchmarks like these on daily.dev._ ### Why does increasing num_speculative_tokens in speculative decoding sometimes reduce throughput? Acceptance rate drops sharply at later draft positions, so beyond a certain proposal length the extra drafting and verification work outweighs the benefit of committing more tokens per round, causing throughput to plateau or regress. For example, DFlash on gemma-4-26B-A4B-it/GSM8K peaked at N=7 (2.70x) but dropped to 2.12x at N=15 as per-position acceptance fell from 88% to single digits at the tail positions. _Tuning speculative decoding parameters gets easier when developers compare acceptance-rate data like this on daily.dev._ ## Similar posts on daily.dev - [Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference](https://daily.dev/posts/co-designing-ai-models-using-speculative-decoding-for-faster-llm-inference-0zr9mkfau) · NVIDIA Developer · 0 upvotes · 0 comments - [Speculative Decoding on vLLM: A Configuration and Decision Framework](https://daily.dev/posts/speculative-decoding-on-vllm-a-configuration-and-decision-framework-wsfye2aez) · DigitalOcean Community · 0 upvotes · 0 comments - [EAGLE-3 Speculative Decoding on AMD Instinct GPUs: Training and Serving with vLLM and AMD Quark](https://daily.dev/posts/eagle-3-speculative-decoding-on-amd-instinct-gpus-training-and-serving-with-vllm-and-amd-quark-x5yf4izas) · vLLM · 0 upvotes · 0 comments - [Speculative Decoding on CPUs: Nearly 4x Faster Token Generation with DFlash](https://daily.dev/posts/speculative-decoding-on-cpus-nearly-4x-faster-token-generation-with-dflash-suiryhqim) · Towards Data Science · 0 upvotes · 0 comments - [How speculative decoding delivers faster LLM inference](https://daily.dev/posts/how-speculative-decoding-delivers-faster-llm-inference-jxnu7qqsj) · Red Hat Developer · 1 upvotes · 0 comments --- Tags: [#amd](https://daily.dev/tags/amd), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/exploring-speculative-decoding-in-vllm-on-amd-gpus-3xgpvjp6n) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Exploring Speculative Decoding in vLLM on AMD GPUs","url":"https://daily.dev/posts/exploring-speculative-decoding-in-vllm-on-amd-gpus-3xgpvjp6n","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/exploring-speculative-decoding-in-vllm-on-amd-gpus-3xgpvjp6n"},"datePublished":"2026-08-23T02:05:56.348Z","dateModified":"2026-09-13T21:59:48.236Z","description":"A detailed technical walkthrough of speculative decoding in vLLM running on AMD Instinct MI300X and MI355X GPUs with ROCm. It covers five drafting approaches...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3791a2c824b3cec72feecde8694051d6?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3791a2c824b3cec72feecde8694051d6?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"vLLM","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"vLLM","logo":"https://media.daily.dev/image/upload/s--hTxEuls9--/f_auto/v1744613054/logos/vllm","url":"https://daily.dev/sources/vllm"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/exploring-speculative-decoding-in-vllm-on-amd-gpus-3xgpvjp6n","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"amd,ai-inference,vllm","timeRequired":"PT67M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"vLLM","item":"https://daily.dev/sources/vllm"},{"@type":"ListItem","position":3,"name":"Exploring Speculative Decoding in vLLM on AMD GPUs"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/exploring-speculative-decoding-in-vllm-on-amd-gpus-3xgpvjp6n#faq","mainEntity":[{"@type":"Question","name":"How do I configure speculative decoding in vLLM with DFlash?","acceptedAnswer":{"@type":"Answer","text":"Set the speculative-config method to dflash and point the model field to a matching DFlash speculator checkpoint, along with num_speculative_tokens for the proposal length. DFlash predicts a whole block of masked positions in parallel using an anchor token, converting fused target-model hidden states into extra key and value information available in every draft-network layer. Many DFlash checkpoints are trained with a fixed block size, and N=7 was frequently among the higher-throughput settings across tested models. Anyone tuning speculative decoding configs can track method-specific setup details like these on daily.dev."}},{"@type":"Question","name":"What throughput speedup does EAGLE-3 speculative decoding achieve versus DFlash on the same model?","acceptedAnswer":{"@type":"Answer","text":"On gemma-4-26B-A4B-it, EAGLE-3 measurements ranged from 2.11x to 2.27x across four datasets, while DFlash reached higher peaks of 2.87x on MATH500 and 2.79x on HumanEval. On Kimi-K2.5, EAGLE-3 reached up to 2.33x while DFlash reached up to 2.68x. DFlash's advantage generally grows with longer proposal lengths (commonly N=7), whereas EAGLE-3's sequential drafting favors shorter proposals like N=4. Comparing speculative decoding methods before picking one is easier when tracking benchmarks like these on daily.dev."}},{"@type":"Question","name":"Why does increasing num_speculative_tokens in speculative decoding sometimes reduce throughput?","acceptedAnswer":{"@type":"Answer","text":"Acceptance rate drops sharply at later draft positions, so beyond a certain proposal length the extra drafting and verification work outweighs the benefit of committing more tokens per round, causing throughput to plateau or regress. For example, DFlash on gemma-4-26B-A4B-it/GSM8K peaked at N=7 (2.70x) but dropped to 2.12x at N=15 as per-position acceptance fell from 88% to single digits at the tail positions. Tuning speculative decoding parameters gets easier when developers compare acceptance-rate data like this on daily.dev."}}]} ```