GSM8K — AI Glossary

Reference details

Related terms

• MATH Benchmark
• Chain of Thought (CoT)
• Reasoning (LLM)

How it works

1. Problems are grade-school word problems requiring several arithmetic steps, written in natural language so the model must extract the quantities and their relationships before computing.
2. Each problem has a single numeric answer, and grading compares that number — so the reasoning is unscored and only the final value counts.
3. Reference solutions include worked steps, which is what made the dataset useful for training and evaluating intermediate reasoning rather than just answers.
4. Chain-of-thought prompting produces large gains, because the task is multi-step and producing the steps is what keeps the intermediate quantities available.
5. Final-answer grading has a known consequence: a correct number reached by faulty reasoning scores the same as a sound derivation, so the metric does not measure what the reference solutions describe.
6. The arithmetic is deliberately elementary, so the difficulty is in reading the problem and sequencing the operations rather than in the computation itself.

Example: Why multi-step problems punish per-step reliability

Take a problem requiring 8 dependent steps, where getting the final number right requires every step to be right. At 95% per-step reliability the whole problem succeeds with probability 0.95^8 = 0.663, so a third of such problems fail despite each individual step being right 19 times in 20. At 90% per step it is 0.9^8 = 0.430, and at 99% it is 0.99^8 = 0.923. Inverting it is the more useful direction: to solve 90% of 8-step problems, per-step reliability must be 0.9^(1/8) = 0.987, nearly 99%. This is why a benchmark of elementary arithmetic separates models at all — the arithmetic is trivial and the sequencing is not — and why gains from producing intermediate steps are so large: writing each step down keeps the quantities available instead of requiring them to be held implicitly, which raises per-step reliability where the compounding is most punishing. It also explains the ceiling effect: once per-step reliability is high, 0.998^8 = 0.984 and the set stops discriminating.

Commonly confused with

This measures multi-step arithmetic reasoning on elementary problems, not mathematical ability. The arithmetic involved is primary-school level; what is tested is reading comprehension, quantity tracking and operation sequencing. Competition-level mathematics benchmarks test something different — insight, proof, non-obvious strategy — and a model strong here can be weak there. The second distinction is from calculator-augmented performance: giving the model a tool changes the task from arithmetic-under-compounding to translation-and-delegation, which is both easier and more representative of how such problems should be solved in production.

When to use it

Reach for it when:

• Assessing multi-step reasoning on models where the set has not saturated, which is where it still discriminates.
• Measuring the effect of intermediate-reasoning prompting, which this task's structure makes unusually visible.
• As a cheap smoke test for quantity tracking, where failure is informative and success is not.
• With the prompting protocol stated, since intermediate reasoning moves the number more than most model differences.

Reach for something else when:

• As a measure of mathematical ability, which competition-level benchmarks address and this does not.
• At the frontier, where saturation leaves differences inside the noise.
• Reading final-answer accuracy as evidence about reasoning quality, since the metric does not grade the steps.
• Without noting whether tool use was permitted, which changes the task rather than improving performance at it.

Primary source

• Cobbe et al., Training Verifiers to Solve Math Word Problems (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to GSM8K — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path