How to Make LLMs 3X Faster
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Speculative decoding is explained as a technique that exploits spare GPU compute during LLM token generation to speed up inference 2-3x without changing output quality. The piece covers why autoregressive decoding is memory-bandwidth bound, how a small draft model proposes candidate tokens that a large target model verifies in a single parallel forward pass, the accept/reject mechanism that guarantees statistically identical output to standard decoding, and how acceptance rate varies by workload (high for code/summarization, low for creative writing). It also surveys four sources of draft tokens (separate small model, extra prediction heads, quantized self-drafting, and text-based lookup), and notes that gains shrink under high concurrency, citing DeepSeek-V3's reported 80-90% acceptance rate and ~1.8x throughput gain, plus a benchmark showing 1.96x speedup at batch size 1 dropping to 1.21x at batch size 128.
Table of contents
What is loop engineering? (Sponsored)Autoregressive DecodingMemory BandwidthParallel VerificationDraft and VerifyLossless GuaranteeAcceptance RateCandidate or Draft SourcesConcurrency LimitsConclusionQuestions this post answers
What is speculative decoding and how does it make LLM inference faster?
Speculative decoding uses a small draft model to generate several candidate tokens ahead of time, then has the large target model verify all of them in a single forward pass instead of one pass per token. Because GPU compute utilization during generation is only 20-40% due to memory bandwidth bottlenecks, this spare capacity can be used to evaluate multiple tokens at once, yielding 2-3x faster generation while producing text statistically identical to standard decoding. daily.dev surfaces practical explainers like this for engineers optimizing LLM serving throughput.
Why is a single LLM token generation step so slow on a GPU?
Generating one token requires transferring the entire set of model weights from GPU memory into the compute units, which for a 70-billion-parameter model at 16-bit precision amounts to roughly 140 GB per token. The actual arithmetic performed on those weights is small in comparison, so compute utilization drops to 20-40% during generation, versus 90-95% during prompt processing, since the GPU spends most of its time moving data rather than calculating. Engineers tuning inference latency track these GPU bottleneck breakdowns on daily.dev.
What acceptance rate did DeepSeek report for speculative decoding on DeepSeek-V3?
DeepSeek reported acceptance rates between 80 and 90 percent for the second predicted token during production serving of DeepSeek-V3, translating to roughly 1.8x generation throughput. DeepSeek trained extra prediction heads during pretraining and reused them at inference as the draft source, noting the approach slightly reduces throughput while significantly improving end-to-end generation latency. Teams comparing inference optimization techniques follow real-world benchmarks like this on daily.dev.