Speculative decoding is an LLM inference optimization that uses a small, fast 'speculator' model to draft multiple tokens ahead, while the larger production model verifies them in a single parallel forward pass. When the speculator guesses correctly (50–80% of the time for predictable tasks), you get 3–5 tokens for the cost of one forward pass, with no loss in output quality. The technique works best for code generation, structured outputs (JSON, SQL), and template-based tasks at low batch sizes. A concrete vLLM implementation is shown using Red Hat AI speculator models, with benchmarks showing ~145 vs ~424 tokens/second on an H100 GPU. Key tuning parameters include number of speculative tokens (start at 4–5) and acceptance rate (target 60–80%). The approach also reduces cost per token by ~60% on the same hardware.
