BIG-Bench Hard (BBH) — AI Glossary

Reference details

Related terms

• MMLU (Massive Multitask Language Understanding)
• ARC Challenge
• Chain of Thought (CoT)

How it works

1. The suite is a filtered subset of a much larger benchmark collection, keeping only tasks on which models of the time scored below a stated threshold relative to human raters.
2. That construction is the defining property: difficulty is defined relative to a particular generation of models, so the set measures what was hard then rather than what is hard now.
3. The retained tasks span deduction, causal judgement, symbolic manipulation and formal-language problems — categories chosen by what survived the filter rather than by a taxonomy of reasoning.
4. Each task is scored separately and the headline figure is an average across them, which means the aggregate weights every task equally regardless of its size or variance.
5. Chain-of-thought prompting produces unusually large gains here, because the retained tasks are mostly multi-step and the filter selected for exactly the kind of problem that intermediate reasoning helps.
6. Per-task example counts are small, so a single task's score carries a wide interval and only the aggregate has useful precision.

Example: Why per-task numbers should not be compared and the average can be

Take 23 tasks of roughly 250 examples each — about 23 x 250 = 5,750 examples in total. For one task near 50% accuracy the standard error of a proportion is sqrt(0.5 x 0.5 / 250) = sqrt(0.001) = 0.0316, about 3.2 points, so the standard error on a per-task difference between two models is sqrt(2) x 3.2 = 4.5 points. Any single-task gap under about 5 points is noise. The aggregate behaves much better: averaging 23 independent task estimates divides the standard error by sqrt(23) = 4.8, giving roughly 3.2 / 4.8 = 0.67 points, and about 0.9 points on a difference. So the suite resolves aggregate differences of one or two points and per-task differences of five or more — a factor of about 4.5 / 0.9 = 5 between the two. The practical consequence is that a table of 23 per-task scores invites exactly the comparison the data cannot support, and the one number worth reading from it is the mean.

Commonly confused with

This measures reasoning on tasks selected for being hard for a prior model generation, which is not the same as measuring reasoning difficulty in general. As models improve, a difficulty-filtered set saturates and stops discriminating — and it saturates unevenly, with the easiest of the retained tasks going first, so the aggregate keeps moving while the informative tasks are exhausted. The related confusion is with broad knowledge benchmarks: those sample subject coverage, while this samples reasoning patterns, so a model can be strong on one and weak on the other and the two are not substitutes.

When to use it

Reach for it when:

• As a robustness check alongside a broad knowledge benchmark, since the two measure different things.
• Assessing whether chain-of-thought or extended reasoning helps, where this suite's composition makes the effect visible.
• Reading the aggregate, which has roughly five times the resolution of any individual task.
• On models in the range where the set has not saturated, which is where it still discriminates.

Reach for something else when:

• Comparing per-task scores, where a gap under five points is inside one standard error.
• As a general capability measure, since the task selection is a historical artefact of one model generation's weaknesses.
• At the frontier, where saturation makes the remaining differences small and unevenly distributed.
• Without stating the prompting protocol, since chain-of-thought moves these numbers more than most model differences do.

Primary source

• Suzgun et al., Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to BIG-Bench Hard (BBH) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path