LLM-as-Judge — AI Glossary

Reference details

Related terms

• Eval (Production Evaluation)
• Evaluation Benchmark
• G-Eval

How it works

1. A rubric is written stating the criteria and the scale explicitly, since a judge asked for a general quality score returns a number whose meaning cannot be reconstructed.
2. The judge receives the output under evaluation, the original input, and a reference answer where one exists, and is asked to emit a verdict against that rubric.
3. Asking for reasoning before the score, rather than after, changes the result — the score becomes conditioned on the stated reasoning instead of justifying a figure already chosen.
4. Pairwise comparison is generally used in preference to absolute scoring, because a model is more consistent at judging which of two outputs is better than at placing one on a scale.
5. Each pair is presented twice with the order reversed and the two verdicts compared, which both cancels positional bias and measures how much of it there is.
6. Several samples are then aggregated by majority vote or mean, which reduces the random component of the verdict but not its bias: averaging shrinks variance as one over the square root of the sample count, while a judge that systematically prefers longer answers prefers them in every sample.

Example: The biases are measurable, so measure them

Present the same pair twice with the order swapped. If the judge picks whichever came first, you have positional bias and your scores are partly noise. The same test catches verbosity preference and self-preference. A judge you have not tested this way is an instrument of unknown accuracy.

Commonly confused with

A judge score is a model's opinion, not a measurement. It is reproducible enough to track direction over time, which is genuinely useful, but it is not ground truth and it moves when the judge model changes. Validate against human labels at least once before trusting it.

When to use it

Reach for it when:

• Quality is a judgement call rather than a checkable answer
• You need evaluation at a volume humans cannot cover
• Pairwise comparison is enough — it is more reliable than absolute scoring

Reach for something else when:

• Correctness is programmatically checkable; check it instead
• The judge has never been validated against human labels
• The judge is grading its own family's output, where self-preference is strongest

Referenced by

This term is used by the following ModelRefs references:

• offline eval stack

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LLM-as-Judge — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path