GPQA Diamond — AI Glossary

Reference details

Related terms

• Evaluation Benchmark
• MMLU (Massive Multitask Language Understanding)

How it works

1. Questions are written by domain PhDs in physics, chemistry and biology, at a level where answering requires actual subject knowledge rather than reading comprehension.
2. Validation is adversarial by design. A question is kept only when other experts in the field answer it correctly and non-experts, given unrestricted web access and substantial time, do not. That second condition is what 'Google-proof' means operationally.
3. The Diamond subset keeps the questions where expert agreement was strongest and non-expert success weakest, which makes it the hardest and most-cited split — and a small one, which has consequences for how precisely it can measure anything.
4. The format is multiple choice, which makes grading unambiguous and installs a chance floor: with four options, a model that knows nothing scores 25% by construction.
5. Scores are therefore read against that floor rather than against zero, since the interesting quantity is how far above chance a result sits, not the raw percentage.
6. Contamination is a standing concern for any public set, so results are most meaningful with the evaluation protocol stated — prompt format, whether reasoning was elicited, and how many samples were taken.

Example: Reading a four-choice score against its floor and its sample size

With four options the chance floor is 1/4 = 25%. A raw score of 50% is therefore not halfway to competence: the chance-corrected position is (0.50 - 0.25) / (1 - 0.25) = 0.25 / 0.75 = 0.333, a third of the way from guessing to perfect. A raw 40% corrects to (0.40 - 0.25) / 0.75 = 0.20, and a raw 70% to (0.70 - 0.25) / 0.75 = 0.60 — so two models whose raw scores differ by a factor of 70 / 40 = 1.75 are in fact 0.60 / 0.20 = 3 times apart in distance above chance. Then there is sample size. The Diamond split is 198 questions, so the standard error of a proportion near 0.5 is sqrt(0.5 x 0.5 / 198) = 0.0355, about 3.6 points, and the standard error on the difference between two models is sqrt(2) x 3.6 = 5.0 points. A 5-point gap on this set is one standard error — indistinguishable from sampling noise. Both corrections point the same way: small raw differences on a small four-choice set carry far less information than a leaderboard ordering implies.

Commonly confused with

GPQA measures expert-level scientific reasoning, not general intelligence or practical research capability, and the Google-proof property is narrower than it sounds. It means a non-expert with a search engine could not answer, which makes the questions resistant to retrieval of a stated answer; it does not make them resistant to a model having seen the question itself, since the set is public. It is also not a measure of whether a model is useful for science: the questions are self-contained multiple choice, while real scientific work involves open problems, no answer options, and the question of what to ask.

When to use it

Reach for it when:

• Comparing frontier reasoning capability, where easier benchmarks have saturated and no longer separate models.
• Assessing whether extended reasoning or sampling strategies help, since the questions are hard enough for the difference to show.
• As one input among several, read in chance-corrected terms with its sample size in view.
• Checking a model against a domain where expert-level knowledge is genuinely required by your use case.

Reach for something else when:

• As a general capability score. Graduate science questions do not predict coding, writing, instruction following or agentic reliability.
• Comparing results across different protocols — prompt format, reasoning elicitation, sample count — which move the number materially.
• Reading small gaps as real. At 198 questions, a five-point difference is roughly one standard error.
• Uncorrected for chance, which overstates low scores and understates the real distance between models.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to GPQA Diamond — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path