ModelRefs / Speculative Decoding — AI Glossary

Speculative Decoding — AI Glossary

An inference technique where a small draft model proposes tokens that a large verifier model accepts or rejects in parallel.

Overview

Speculative decoding achieves 2–4× throughput at zero quality loss when the draft acceptance rate is high. Used by vLLM, TensorRT-LLM, and most modern inference stacks. The draft model is typically 10–20× smaller than the verifier.

Reference details

Topic	inference
Last reviewed	2026-06-24

How it works

A small draft model generates several tokens autoregressively, which is cheap because its own forward passes are a fraction of the target model's.
The target model then scores all of those positions in a single forward pass, since the draft has supplied the tokens that would otherwise have to be generated one at a time.
Each drafted token is accepted with probability equal to the target's probability over the draft's, capped at one — so a token the target liked at least as much as the draft did is always kept.
On the first rejection a replacement is sampled from the difference between the two distributions, normalised, rather than from the target distribution directly.
That residual sampling is what makes the procedure exact: the combination of accepting by ratio and resampling from the remainder is provably distributed identically to sampling from the target alone.
So the method is lossless by construction rather than by measurement, and the only thing a poorly matched drafter costs is throughput — the output distribution is unchanged either way.

Example: Count the tokens you get per expensive pass

A small draft model proposes 4 tokens; the large model verifies them in one pass. If each proposed token is accepted with probability 0.8, the expected run before the first rejection is 0.8 + 0.8² + 0.8³ + 0.8⁴ = 0.8 + 0.64 + 0.512 + 0.41 ≈ 2.36, and the verifier contributes one correct token regardless — roughly 3.4 tokens per expensive pass instead of 1. The critical property is that the output distribution is provably unchanged: rejected drafts are resampled from the large model, so this is not an approximation and costs no quality. The risk is the other direction — a poorly matched draft model with a low acceptance rate makes you pay for drafting and get almost nothing back.

Commonly confused with

Speculative decoding is lossless, which separates it from quantization, distillation and early exit — all of which trade quality for speed. It also is not batching: batching raises aggregate throughput while individual requests get no faster, whereas speculation reduces latency for a single request. The two compose, but they solve different problems.

When to use it

Reach for it when:

Latency-sensitive single-request serving, where batching cannot help
Where a small model from the same family and tokenizer is available as the drafter
Predictable text — code and structured output tend to have high acceptance rates

Reach for something else when:

Throughput-bound batch serving, where the GPU is already saturated
Without measuring the acceptance rate on your own traffic — below roughly break-even it is a net loss
Mismatched draft and target vocabularies, which breaks the verification step

Referenced by

This term is used by the following ModelRefs references:

become llm engineer

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Speculative Decoding — AI Glossary.

Frequently asked questions

What is Speculative Decoding?