Chain of Thought (CoT) — AI Glossary

Reference details

Related terms

• Zero-Shot Prompting
• Few-Shot Prompting
• Reasoning Model
• Self-Consistency
• Tree of Thoughts (ToT)

How it works

1. A forward pass has fixed depth, so the computation available between reading a prompt and emitting one token is bounded no matter how hard the problem is.
2. Every token the model emits is appended to its own context, which means the next token is computed with the previous one as an input rather than as something held internally.
3. Writing an intermediate result therefore converts depth into sequence length: a quantity the model would have had to hold implicitly becomes text it can attend to directly.
4. This is why the gain appears on problems with genuine intermediate state and not on lookup — there is nothing to externalise when the answer is one step away.
5. Prompting for it works because instruction tuning included worked examples, so the model has a learned format for stepwise output and the trigger phrase selects it.
6. Training for it instead means rewarding traces that reach verifiable answers, which produces a model that allocates its own trace length per problem rather than following a prompt's instruction to do so.

Example: Where the reasoning has to go

“A train leaves at 14:20 and takes 3h50m; the connection departs 15 minutes after arrival. When?” Answered directly, models often slip on the hour boundary. Asked to work through it — 14:20 + 3h50m = 18:10, plus 15 minutes = 18:25 — the intermediate step is written down, so the arithmetic is done rather than recalled, and a wrong step is visible instead of hidden.

Commonly confused with

The visible chain is not necessarily the computation. Models can produce a correct-looking rationale for a wrong answer, and a plausible rationale for a right one reached another way. Chain-of-thought improves accuracy on multi-step problems; it is not an explanation of what the network did.

When to use it

Reach for it when:

• The task genuinely has intermediate steps: arithmetic, multi-hop, planning
• You want a wrong step to be inspectable rather than buried
• The model is not already a reasoning model with this built in

Reach for something else when:

• Single-step lookup or classification, where it costs tokens and adds nothing
• Latency-critical paths — reasoning tokens are generated before any answer appears
• On reasoning models trained to reason internally; prompting for it can interfere

Primary source

• Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (arXiv)

Referenced by

This term is used by the following ModelRefs references:

• chain of thought
• self consistency
• tree of thoughts
• learn prompt engineering
• master dspy

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Chain of Thought (CoT) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path