Greedy Decoding — AI Glossary

Reference details

Related terms

• Beam Search
• Sampling (LLM)
• Temperature

How it works

1. The model produces a logit vector over the whole vocabulary for the next position, one score per token.
2. Greedy decoding takes the argmax and emits it. No sampling step is required — the ranking alone determines the choice, which is why temperature has no effect on greedy output.
3. The chosen token is appended to the context and the forward pass repeats. Each step commits irrevocably: there is no beam to hold alternatives and no backtracking.
4. Because the choice is deterministic given identical logits, the same prompt and the same weights reproduce the same output — subject to floating-point non-determinism from batching and kernel selection, which is why 'deterministic' holds more cleanly in theory than on a busy GPU.
5. The failure mode is structural: committing to the locally best token at every step does not maximise the probability of the whole sequence, so a high-probability continuation that requires a low-probability first token is unreachable.
6. Repetition arises from the same mechanism. Once a phrase becomes the most likely continuation of itself, the argmax loops, which is why repetition penalties exist as a patch rather than a fix.

Example: Why locally best is not globally best

Two continuations are available at a branch point. Path A starts with a token of probability 0.6 and continues with tokens of probability 0.5 and 0.5, giving 0.6 x 0.5 x 0.5 = 0.15. Path B starts with a token of probability 0.4 and continues with 0.9 and 0.9, giving 0.4 x 0.9 x 0.9 = 0.324. Greedy decoding takes the 0.6 token because it is the argmax at step one, and ends on the sequence that is less than half as likely overall: 0.15 against 0.324. This is the gap beam search exists to close, and it is why greedy is the right default for extraction — where one token is overwhelmingly correct and no such branch exists — and the wrong default for open generation, where branches like this are everywhere.

Commonly confused with

Greedy decoding is not temperature 0, although the two produce the same output and are used interchangeably. Temperature scales logits before sampling; as temperature approaches 0 the sampling distribution collapses onto the argmax, so temperature 0 is implemented as greedy in most runtimes precisely because the limit is undefined by division. The distinction matters when reading an API: temperature 0 is a sampling parameter a provider may implement with a floor, while greedy is an algorithm with no parameter at all. Neither guarantees bit-identical output across requests, because batching changes reduction order in floating point.

When to use it

Reach for it when:

• Structured extraction and classification, where exactly one output is correct and creativity is a defect.
• Code generation against an exact format requirement, where a plausible alternative token is a syntax error.
• Debugging and regression testing, where you need the same prompt to produce the same output as far as the hardware allows.
• Latency-sensitive paths, since greedy requires no sampling step and no additional forward passes.

Reach for something else when:

• Open-ended generation — prose, brainstorming, dialogue — where the degenerate repetition loop is the normal outcome rather than an edge case.
• Any workflow that samples several candidates and ranks them, since greedy returns the same candidate every time and the ensemble collapses to one.
• Tasks where the best answer needs an unlikely opening move, such as a proof that starts with a non-obvious lemma.
• As a determinism guarantee in a contract or audit trail. Floating-point non-determinism under varying batch composition can change output even under greedy.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Greedy Decoding — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path