MT-Bench — AI Glossary

Reference details

Related terms

• AlpacaEval
• Chatbot Arena
• LMSYS

How it works

1. A fixed set of multi-turn prompts spans several categories, so a score reflects breadth rather than one skill.
2. Each item has two turns, and the second depends on the first — which is what makes it a conversation test rather than a prompt test.
3. Responses are scored by a strong model acting as judge against a rubric, because the answers are open-ended and have no single reference.
4. The judge's agreement with human preference is measured and reported, so the metric's own reliability is part of the published method.
5. Being fixed and public makes it reproducible and contaminable, the same trade every static benchmark carries.

Example: What the second turn catches

A model answers turn one well and, asked in turn two to revise under a constraint, ignores the constraint and restates its first answer. Single-turn scoring records a good answer and the failure never appears. Across a suite, a model can be strong on turn one and systematically weak on turn two — instruction retention is a separate capability from instruction following, and only a multi-turn format measures it. That is the design intent rather than a side effect.

Commonly confused with

Conflated with crowd-sourced arena rankings from the same group, and the methods are opposite. This is a fixed prompt set scored by a model judge, so it is reproducible and can be contaminated. An arena collects human preference votes on prompts users chose, so it resists contamination and cannot be replicated exactly. Both measure preference-like quality rather than task correctness, and a model-judge score inherits the judge's biases — including a known tendency to favour longer and more fluent answers.

When to use it

Reach for it when:

• Comparing instruction-following and multi-turn coherence on a reproducible basis.
• A standardised judge-scored benchmark is wanted across categories.
• Explaining why multi-turn evaluation differs from single-turn.

Reach for something else when:

• Factual correctness is the question, which a preference rubric does not measure.
• Contamination is the concern, since the prompts are public and fixed.
• As evidence of performance on your own workload.

Primary source

• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to MT-Bench — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path