LMSYS — AI Glossary

Reference details

Related terms

• Chatbot Arena
• AlpacaEval
• Open LLM Leaderboard

How it works

1. Chatbot Arena serves a visitor two anonymous models side by side for the same prompt, so the comparison is blind in both directions.
2. The visitor votes for one response. Model identities are revealed only afterwards, which is what keeps brand preference out of the signal.
3. Votes are pairwise comparisons, not scores, so they are fitted with a Bradley-Terry model to produce a rating per model on a common scale.
4. Because prompts come from whoever shows up rather than from a fixed set, the query distribution is natural and uncontrolled — the source of both the method's strength and its sampling bias.
5. The same group ships the surrounding infrastructure — a serving stack and a structured-generation runtime — which is why the name appears in deployment contexts unrelated to ranking.

Example: What a pairwise vote can and cannot establish

Two models are compared 400 times and one wins 220. That is 55%, and the standard error on a proportion at n=400 is sqrt(0.55 × 0.45 / 400) ≈ 0.025, so about 2.5 points: the 55% is roughly two standard errors from 50% and the ordering is probably real. The same 55% at n=40 carries an error near 8 points and establishes nothing. This is why a leaderboard's newest entries move and then settle — the rating is a measurement with a sample size, not a property of the model.

Commonly confused with

Used as a synonym for the Arena, which is one project of several from the same group. LMSYS is the organisation; Chatbot Arena is its ranking platform, and FastChat and SGLang are serving and structured-generation infrastructure that have nothing to do with ranking. When a deployment document mentions LMSYS it usually means the runtime, and when an evaluation document does it usually means the Arena — the word alone does not distinguish them.

When to use it

Reach for it when:

• Attributing the Arena ranking methodology or the serving tools to their source.
• Explaining why a crowd-sourced preference ranking is constructed differently from a static benchmark.

Reach for something else when:

• As the name of the leaderboard itself when precision matters — that is Chatbot Arena.
• As a benchmark: the organisation publishes rankings, it is not a measurement.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LMSYS — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path