Beam Search — AI Glossary

Reference details

Related terms

• Greedy Decoding
• Sampling (LLM)
• Temperature

How it works

1. Several partial sequences are kept at every step rather than one, so a locally poor token that leads somewhere better is not discarded immediately.
2. Each candidate is extended by every plausible next token, and the resulting set is pruned back to the beam width by cumulative score.
3. Scores are summed log-probabilities, which systematically favours shorter sequences — so a length penalty is normally applied to compensate.
4. The output is the highest-scoring complete sequence, which makes the method deterministic given the same inputs.
5. That determinism is also its weakness for open-ended text: maximising likelihood produces generic, repetitive output rather than interesting output.

Example: Why it won translation and lost open generation

A beam of 4 costs roughly four times the compute of greedy decoding per step, and for translation that buys a measurably better sequence because there is approximately one right answer to find. For open-ended writing the highest-likelihood sequence is the blandest one — the model's safest continuation at every step — so four times the compute buys worse-reading text than sampling. The method did not get worse; the task changed from finding one answer to producing a good one.

Commonly confused with

Set against sampling as though one were simply better, when they optimise different things. Beam search searches for the most likely sequence and is deterministic; sampling draws from the distribution and is varied. Likelihood and quality coincide for constrained tasks like translation and diverge for open generation, which is why modern chat decoding samples with nucleus or temperature controls instead. It is also not speculative or lookahead decoding: those accelerate production of the same output distribution, while this changes which sequence is produced.

When to use it

Reach for it when:

• Constrained generation with approximately one correct output, such as translation or transcription.
• Deterministic output is required for reproducibility.
• The extra compute per step is affordable and buys a measurably better sequence.

Reach for something else when:

• Open-ended generation, where maximising likelihood produces generic text.
• Latency or cost is tight, since the beam multiplies per-step compute.
• Variety is wanted, which a deterministic search cannot provide.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Beam Search — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path