ModelRefs / Evaluation Benchmark — AI Glossary
Evaluation Benchmark — AI Glossary
A standardized dataset and scoring protocol for measuring model capability on a specific task or domain. A benchmark is not an evaluation of your system.
Overview
Benchmarks (MMLU, GPQA, SWE-Bench, HumanEval) enable cross-model comparison. Risks: contamination (test data in training), saturation (frontier models max out), and gameability (models tuned to specific benchmarks). Production evals on private data remain essential alongside public benchmarks.
Reference details
| Topic | evaluation |
|---|---|
| Also known as | benchmark, eval benchmark |
| Last reviewed | 2026-06-24 |
Related terms
How it works
- A benchmark fixes a dataset with declared splits, a prompting protocol, and a scoring function, and it is the combination rather than the dataset alone that makes scores comparable.
- Submissions are run under that protocol and reported on a leaderboard, which is what creates the shared reference point and also the incentive to optimise against it.
- Contamination is checked by searching training corpora for overlap with test items, or by seeding the dataset with canary strings that a trained model will reproduce if it saw them.
- Neither check is conclusive for a model whose training data is undisclosed, which is why contamination is usually argued from publication dates rather than demonstrated.
- Saturation is diagnosed by comparing the spread between leading systems against the dataset's own label-error rate: once the gap is smaller than the noise, the ranking is measuring annotation mistakes.
- Both failure modes develop over time rather than being present at release, so a benchmark has a useful life and the question about any given one is how much of it remains.
Example: Saturation and contamination, concretely
A benchmark where frontier models all score 88-91% has stopped discriminating — the remaining gap is mostly noise and label error. Separately, if the test set predates the model's training cutoff and is public, some of it is plausibly in the training data. Both problems make a headline number look more meaningful than it is.
Commonly confused with
A benchmark is not an evaluation of your system. It measures a model on a shared public task under fixed conditions. Your eval measures your pipeline on your data against your criteria — and a model that leads a leaderboard can still lose on the one that governs your release.
When to use it
Reach for it when:
- Shortlisting models before committing to your own evaluation
- Tracking capability trends across releases
- You need a shared reference others recognise
Reach for something else when:
- As a release gate — it does not measure your task
- When the benchmark is saturated or plausibly contaminated
- Comparing scores from different harnesses or few-shot settings
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Evaluation Benchmark — AI Glossary.
Frequently asked questions
What is Evaluation Benchmark?
A standardized dataset and scoring protocol for measuring model capability on a specific task or domain.
Is Evaluation Benchmark the same as benchmark?
Yes — benchmark, eval benchmark are common aliases for Evaluation Benchmark.
What concepts are related to Evaluation Benchmark?
Closely related concepts include eval, llm as judge, mmlu, swe bench.