Temperature — AI Glossary

Reference details

Related terms

• Top-P (Nucleus Sampling)
• Inference

How it works

1. The model's final layer emits a logit for every token in the vocabulary, which is an unnormalised score and not yet a probability.
2. Every logit is divided by the temperature before the softmax, which is the entire mechanism — nothing else about the model's computation changes.
3. Because softmax exponentiates, dividing the logits rescales the gaps between them multiplicatively: a temperature below 1 widens the gaps and concentrates probability on the leaders.
4. A temperature above 1 narrows the gaps, lifting the tail of the distribution, which is why unlikely tokens become reachable rather than merely more likely.
5. As temperature approaches zero the largest logit's share approaches one, so sampling degenerates into taking the maximum — the greedy decode.
6. A token is then drawn from the rescaled distribution, usually after top-k or top-p has truncated it, which means temperature and the truncation parameters interact rather than acting independently.

Example: Zero is not quite deterministic, and the odds compound

Temperature 0 selects the highest-probability token each step, which is as close to reproducible as an API gets — but batching, hardware and floating-point non-associativity can still shift a tie. Treat 0 as strongly repeatable, not as a guarantee, and pin a seed where the provider offers one. The reason the guarantee fails is that reproducibility is per token and the response is a sequence of them. Suppose only 0.2% of positions have a top-two logit gap small enough for summation order to flip it. Over a 500-token response the chance that no position flips is 0.998^500 ≈ 0.368, so the chance that at least one does is about 63.2% — and one flipped token changes every token after it. A per-step property that holds 99.8% of the time therefore produces identical output on barely a third of responses.

Commonly confused with

Temperature is not a creativity dial. It rescales the probability distribution: higher values raise the chance of low-probability tokens, which reads as creative when it works and as incoherent when it does not. It does not add ideas, only variance. It is also not the only parameter shaping the draw — top-k and top-p truncate the distribution after temperature has rescaled it, so raising temperature widens a field that top-p then trims, and tuning one without the other rarely does what is expected.

When to use it

Reach for it when:

• Low for extraction, classification and anything with one right answer
• Higher for drafting, brainstorming or generating varied synthetic data
• Higher when you sample several outputs and vote across them
• At zero with a pinned seed when you need the closest thing to reproducibility an API offers

Reach for something else when:

• Raising it to fix dull output — that is usually a prompt problem
• Non-zero while debugging, where reproducibility matters more
• Tuning it before the prompt and the schema are settled
• Relied on at zero for bit-identical long outputs, which held for barely a third of responses above

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Temperature — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path