Open LLM Leaderboard — AI Glossary

Reference details

Related terms

• MMLU (Massive Multitask Language Understanding)
• LMSYS
• BigCodeBench

How it works

1. Submitted open-weight models are evaluated on a fixed task set with a shared harness, so numbers are produced the same way for every entry.
2. That standardisation is the point: scores self-reported by publishers use different prompts and parsing and are not comparable.
3. Tasks are chosen to resist saturation, and the set has been revised as older benchmarks became too easy to separate models.
4. Results are reproducible because the harness and configuration are public, which is what distinguishes it from a preference ranking.
5. Fixed public datasets are contaminable, so a high score can reflect training exposure rather than capability — the structural weakness of any static benchmark.

Example: Why standardised harnesses change the ordering

Two publishers report the same benchmark: one scores with a tailored prompt and lenient answer parsing, the other with a plain prompt and strict matching. The gap between those methodologies can exceed the gap between the models, so self-reported numbers can order them backwards. Running both through one harness removes the methodology variable and leaves the model difference. The leaderboard's value is the fixed denominator, not the particular tasks.

Commonly confused with

Grouped with preference rankings as though both measured the same thing. This scores models against fixed questions with known answers and is reproducible; a crowd-sourced arena measures which output humans prefer on prompts they chose, and is not reproducible but is far harder to contaminate. The two disagree routinely, and that is informative rather than a defect — one measures task capability, the other measures how the output reads. Neither substitutes for evaluation on your own workload.

When to use it

Reach for it when:

• Comparing open-weight models on a standardised, reproducible basis.
• Explaining why self-reported benchmark numbers are not comparable across publishers.
• Building a shortlist before testing candidates on your own tasks.

Reach for something else when:

• As evidence of performance on your workload, which it does not measure.
• Where contamination is the concern, since the datasets are public and fixed.
• Judging response quality or style, which a preference ranking addresses.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Open LLM Leaderboard — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path