ModelRefs / Evaluation Harness — AI Glossary

Evaluation Harness — AI Glossary

A standardized framework for running reproducible LLM evaluations across benchmarks with consistent prompting and scoring.

Overview

EleutherAI's LM Evaluation Harness (lm-eval) is the reference: supports 200+ benchmarks (MMLU, ARC, HellaSwag, GSM8K, HumanEval) with standardized few-shot prompting, multiple-choice normalization, and generation-based scoring. Used by HuggingFace Open LLM Leaderboard. Ensures apples-to-apples comparison across model checkpoints.

Reference details

Topic	ecosystem
Last reviewed	2026-06-24

How it works

Each benchmark is declared as a task specification: where the dataset comes from, how a prompt is built from an item, how the answer is extracted, and how it is scored.
Few-shot examples are drawn by the harness from a held-out split using a fixed seed, so the same run requested twice assembles identical prompts.
Prompts are rendered from a stored template rather than written per run, which is what makes a prompt change a recorded configuration change instead of an invisible one.
Model output is passed through a declared extraction rule — a regex, a log-likelihood comparison across choices, or a parser — because the scoring depends entirely on which rule was used.
For multiple-choice tasks the harness commonly scores by comparing the log-likelihood of each option rather than by generating text, and length-normalises those scores.
Results are written alongside the full configuration that produced them, so a reported number carries the task version, few-shot count, template and extraction rule that make it reproducible.

Example: Why the same benchmark gives different numbers, and how many it can give

Two teams report MMLU for the same checkpoint and differ by several points. Neither is lying: few-shot count, prompt template, answer-extraction rule and normalisation all differ. A harness fixes those choices so the comparison is about the model. Without one, benchmark numbers are not comparable across sources. The size of that ambiguity is worth computing rather than describing. Suppose the few-shot count is anywhere from 0 to 5, three prompt templates are in circulation, the answer may be extracted by regex or by comparing option likelihoods, and the likelihood may be length-normalised or not. That is 6 x 3 x 2 x 2 = 72 distinct configurations, every one of which may be reported under the same benchmark name. A harness does not make the model better; it collapses 72 reportable numbers to one.

Commonly confused with

The harness is the runner, not the benchmark. The benchmark is the dataset and scoring rule; the harness standardises prompting, extraction and normalisation around it. Reporting a benchmark score without naming the harness and settings omits most of what makes it reproducible. A harness version is part of that: the same tool updates its templates and extraction rules between releases, so two runs of the same harness on the same checkpoint can differ, and the version belongs in the recorded result alongside the score.

When to use it

Reach for it when:

Comparing checkpoints, quantisations or fine-tunes of your own
Publishing numbers others should be able to reproduce
Tracking a model over time under identical conditions
With the harness version recorded, since its templates change between releases

Reach for something else when:

As a substitute for evaluation on your own task and data
Comparing your harness run against a vendor's differently configured one
Reporting a figure without the few-shot count, template and normalisation, which spanned 72 configurations above
Assuming two runs of the same harness agree across its own versions

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Evaluation Harness — AI Glossary.

Frequently asked questions

What is Evaluation Harness?