Inference — AI Glossary

Reference details

Related terms

• Latency
• Throughput
• Quantization
• Inference Cost

How it works

1. The input text is tokenised and each token mapped to an embedding vector, with positional information added so the stack can distinguish order.
2. That sequence passes through every transformer block in turn, and the final block's output for the last position is projected to a logit per vocabulary token.
3. One token is selected from those logits by the sampling rule in force, appended to the sequence, and the whole forward pass runs again for the next position.
4. Recomputing the earlier positions each time would be quadratic waste, so each block's keys and values are retained in a cache and only the new position is computed.
5. That cache is what makes generation incremental, and it is also what grows with sequence length — per-request memory rises with every token emitted.
6. Generation stops when a stop token is produced, a stop sequence matches, or the token limit is reached, which is why a truncated response and a finished one are distinguished by a finish reason rather than by the text.

Example: Two phases with different bottlenecks, and which one owns the clock

Prefill processes the whole prompt in parallel and is compute-bound — it is why time-to-first-token scales with prompt length. Decode then emits one token at a time, each requiring a pass over the KV cache, and is memory-bandwidth-bound. Optimisations target one phase or the other, which is why a change that improves throughput can leave latency untouched. Which phase owns the wall clock is usually the opposite of what the token counts suggest. Take a 2,000-token prompt and a 100-token answer. If prefill runs at 10,000 tokens a second it takes 2,000 / 10,000 = 0.2 seconds; if decode runs at 50 tokens a second it takes 100 / 50 = 2.0 seconds. Decode is 2.0 / 2.2 ≈ 90.9% of the elapsed time while being under 5% of the tokens processed. Optimising the phase with most of the tokens in it is therefore usually the wrong call.

Commonly confused with

Inference is not the inverse of training. Training computes gradients over batches; inference is forward passes only, dominated by memory bandwidth rather than raw compute. Hardware that trains well does not automatically serve well, and the reverse also holds. The phases are also not interchangeable within inference itself: prefill is compute-bound and decode is bandwidth-bound, so a single figure describing “inference speed” has blended two measurements with different constraints and different remedies.

When to use it

Reach for it when:

• Reasoning about production cost, which inference dominates for most products
• Choosing serving hardware, where bandwidth usually binds before FLOPs
• Diagnosing whether a slowdown is prefill or decode
• When attributing wall-clock time, since decode held 90.9% of it on 5% of the tokens above

Reach for something else when:

• Do not size capacity from training benchmarks; the constraints differ
• Do not treat throughput and latency as one number — they optimise against each other
• Quoted as a single inference-speed figure, which blends two differently constrained phases
• Optimising whichever phase holds most of the tokens, which is usually not the one holding the time

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Inference — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path