AlpacaEval — AI Glossary

Reference details

Related terms

• MT-Bench
• Chatbot Arena
• LMSYS

How it works

1. A fixed instruction set is run through the candidate model, and a reference model's outputs for the same instructions are held constant across every evaluation.
2. A judge model is shown the two responses and asked which is better, with the presentation order randomised per pair so a positional preference averages out rather than accumulating.
3. The raw win rate is the proportion of pairs the candidate took, which is a number about the candidate relative to that one reference and not an absolute score.
4. Length control is a regression rather than a filter: preference is modelled as a function of the length difference alongside a per-model term, and the reported figure is that model term with the length effect held at zero.
5. So the correction removes the component of the win rate attributable to being longer, leaving what remains attributable to the model — which is a statistical adjustment rather than a change to the judging.
6. Everything the regression does not model survives untouched, so style affinity between judge and candidate, formatting preferences and a bias toward confident phrasing all still contribute.

Example: What length control is correcting for

The framework asks a judge model which of two responses is better, comparing a candidate against a fixed reference. Judges have a well-documented bias towards longer answers, so a model can improve its win rate by being more verbose without being more correct. Length-controlled scoring fits away the length effect so the remaining difference is not merely wordiness. That fixes one known bias, and leaves the others: the judge still prefers its own family's style, its own formatting conventions, and confident phrasing over hedged phrasing.

Commonly confused with

This is an automated stand-in for human preference, not a capability benchmark. It measures which response a judge prefers on a fixed instruction set — not correctness, not factuality, and not agentic ability. Read it beside a task benchmark with verifiable answers, and treat a win rate against a reference as relative to that reference.

When to use it

Reach for it when:

• Cheap iteration on instruction-tuning and preference-training ablations
• Ranking candidates before committing to an expensive human preference study
• Tracking one model's changes over time with the judge and reference held fixed

Reach for something else when:

• Evaluating a model from the judge's own family, where style affinity inflates the result
• Any claim about factual accuracy — preference and correctness are different axes
• Comparing numbers produced with different judges, references or scoring variants

Primary source

• Dubois et al., Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to AlpacaEval — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path