MMLU (Massive Multitask Language Understanding) — AI Glossary

Reference details

Related terms

• Evaluation Benchmark
• GPQA Diamond

How it works

1. Each item is a question with four lettered options drawn from one of fifty-seven subjects, and the subject label is what makes the per-domain breakdown possible.
2. The standard evaluation does not ask the model to write an answer: it computes the likelihood the model assigns to each option and takes the highest.
3. That means the measurement is of relative preference among four supplied strings rather than of producing an answer, which is a materially easier task than open generation.
4. Likelihoods are usually length-normalised, because a longer option accumulates more negative log-probability and would otherwise be systematically disfavoured.
5. Few-shot examples are prepended from a development split, and both the number of examples and the order of the options measurably move the result — so neither can be omitted from a reported figure.
6. Because the format supplies the candidate set, a model can also reach the right option by eliminating the other three, which is the shortcut the harder variant was constructed to reduce.

Example: Why the headline number stopped discriminating

Four-choice questions mean random guessing scores about 25%. When frontier models cluster in the high eighties, the remaining spread is a few points sitting on top of known label errors and possible contamination. The benchmark still separates weak models from strong ones; it no longer separates strong from strong.

random baseline = 1 / 4 = 25%
frontier cluster ~ high 80s  -> a few points of usable spread

Commonly confused with

MMLU measures recalled knowledge under multiple choice, not reasoning. The format lets a model succeed by elimination without following an argument — which is exactly what MMLU-Pro was built to reduce by adding options and harder distractors.

When to use it

Reach for it when:

• Coarse capability screening across a wide subject range
• Tracking generational improvement where the gap is large
• You need a benchmark others will recognise

Reach for something else when:

• Separating frontier models, where it has saturated
• As evidence of reasoning ability
• Comparing numbers from different harnesses or few-shot settings

Primary source

• Hendrycks et al., Measuring Massive Multitask Language Understanding (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to MMLU (Massive Multitask Language Understanding) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path