Chatbot Arena — AI Glossary

Reference details

Related terms

• AlpacaEval
• MT-Bench
• LMSYS

How it works

1. A visitor submits a prompt and receives two responses from models they cannot identify.
2. They vote for one, or declare a tie. Identities are revealed after the vote, keeping brand reputation out of the comparison.
3. Votes are pairwise, so a Bradley-Terry fit converts many head-to-head outcomes into a single rating scale on which models never directly compared can still be ordered.
4. Pairings are sampled to favour informative matchups — models close in current rating — because a contest between distant models resolves little.
5. Ratings carry confidence intervals that narrow with votes, which is why new entries move before settling.

Example: Reading a rating gap as a win probability

On an Elo-style scale a 100-point gap corresponds to an expected score of 1 / (1 + 10^(−100/400)) ≈ 0.64, so the higher-rated model is preferred in roughly 64% of comparisons — not in nearly all of them. A 30-point gap gives 1 / (1 + 10^(−30/400)) ≈ 0.54, which is close to a coin flip and within many confidence intervals. This is the arithmetic behind treating small leaderboard gaps as noise: the scale is logistic, and adjacent ranks routinely describe models that beat each other about half the time.

Commonly confused with

Placed alongside benchmarks like MMLU as though it measured the same kind of thing. A static benchmark scores a model against fixed questions with known answers and is reproducible; the Arena measures which output humans prefer on prompts they chose, and is not. That makes it hard to contaminate and impossible to replicate exactly. It also measures preference rather than correctness — response style, length and formatting move votes — so a model can rank highly by being pleasant to read. Both signals are real, they are not substitutes, and the sampling bias of a self-selected voter population applies to one of them.

When to use it

Reach for it when:

• You need a signal about perceived response quality on open-ended prompts.
• Contamination is the concern and a fixed question set cannot answer it.
• Comparing models on natural user prompts rather than curated tasks.

Reach for something else when:

• You need a reproducible number for a specific capability — use a task benchmark.
• The ranking gap is small, where the interval and the logistic scale make the ordering weak evidence.
• Your workload looks nothing like the voter population's prompts.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Chatbot Arena — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path