Draft Model — AI Glossary

Reference details

Related terms

• Speculative Decoding
• Throughput
• Time to First Token (TTFT)

How it works

1. A small model proposes several tokens ahead, generating them autoregressively at a fraction of the large model's per-token cost.
2. The large model then scores all the proposed tokens in a single forward pass, because verifying a known sequence is parallel in a way generating it is not.
3. A rejection-sampling step accepts the longest prefix of proposals consistent with the large model's own distribution and resamples at the first disagreement.
4. That correction is what preserves the output distribution exactly: the result is statistically identical to sampling from the large model alone, which is the property that makes the technique free of quality cost.
5. Throughput therefore depends on the acceptance rate — how often the draft agrees with the target — and on how many tokens are proposed per round.
6. Acceptance depends on how well the draft approximates the target, so a draft from the same family and training data accepts far more often than an unrelated small model of the same size.

Example: Why the acceptance rate dominates the speculation depth

With acceptance probability a per token and k proposals per round, the expected number of tokens produced per target forward pass is (1 - a^(k+1)) / (1 - a). At a = 0.8 and k = 4 that is (1 - 0.8^5) / (1 - 0.8) = (1 - 0.32768) / 0.2 = 3.36 tokens per pass, against 1 without speculation — a 3.36x throughput ceiling before accounting for the draft model's own cost. Now drop the acceptance rate to 0.5 with the same k = 4: (1 - 0.5^5) / 0.5 = (1 - 0.03125) / 0.5 = 1.94, barely half the benefit. Raising k instead does little at the low rate — at a = 0.5 and k = 8 it is (1 - 0.5^9) / 0.5 = 1.996, essentially the same 2x, because the series converges to 1 / (1 - a) = 2 and no depth exceeds it. That asymmetry is the whole engineering lesson: the ceiling is set by 1 / (1 - a), so at 0.8 it is 5 and at 0.5 it is 2, and the way to improve a speculative decoder is a better-matched draft rather than deeper speculation.

Commonly confused with

Speculative decoding is lossless and that is unusual among inference optimisations, which is why it should not be grouped with quantisation or distillation. Quantisation and distillation trade output quality for speed; speculation trades compute on a small model for latency on a large one while the rejection-sampling step guarantees the same output distribution as the target alone. So the risk profile differs in kind: a badly matched draft makes speculation slower than no speculation, and it cannot make the output worse.

When to use it

Reach for it when:

• Latency-sensitive single-stream generation, where the ceiling of 1 / (1 - acceptance) is the available speedup.
• With a draft from the same family and tokenizer as the target, which is what makes acceptance high.
• Where output must be identical to the target model's, since the technique guarantees that and alternatives do not.
• Small batch sizes, where the target model's forward pass is latency-bound rather than throughput-bound.

Reach for something else when:

• Large-batch serving, where the target's passes are already well utilised and the draft competes for the same hardware.
• With a poorly matched draft, where low acceptance makes the overhead exceed the saving.
• When memory is constrained, since both models must be resident simultaneously.
• Expecting deeper speculation to compensate for a low acceptance rate, which the convergence to 1 / (1 - a) rules out.

Primary source

• Leviathan et al., Fast Inference from Transformers via Speculative Decoding (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Draft Model — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path