ModelRefs / G-Eval — AI Glossary
G-Eval — AI Glossary
An LLM-as-judge framework that generates evaluation steps via chain-of-thought before scoring. The auto-generated rubric is a key feature.
Overview
G-Eval (Liu et al., 2023) outperforms vanilla LLM judging on summarization and dialogue tasks by forcing the judge to articulate explicit evaluation criteria before assigning a score. The auto-generated rubric is a key feature.
Reference details
| Topic | evaluation |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
How it works
- An evaluation task and criteria are given to a judge model, which is first asked to produce the evaluation steps rather than a score — so the rubric is generated before anything is assessed.
- Those generated steps become the instruction for the actual scoring pass, which is the framework's central claim: a judge that has articulated what it is looking for scores more consistently than one asked directly for a number.
- Scoring then asks for a rating on a small integer scale, and the response is read as a distribution over the score tokens rather than as a single digit.
- The score is the probability-weighted sum of those candidates, which converts a coarse integer judgement into a continuous value and recovers resolution the discrete scale throws away.
- The method is reference-free: no gold answer is required, which is what makes it usable on open-ended generation where no single correct output exists.
- The judge's own agreement with human raters is the ceiling on everything, so a reported correlation with human judgement is part of the result rather than a footnote to it.
Example: What probability-weighting recovers from a coarse scale
Ask a judge for a rating from 1 to 5 and read only the top token: an output of 4 is 4, and every response between 'barely a 4' and 'almost a 5' collapses to the same number. Now read the distribution. If the judge puts 0.6 on the token 4 and 0.4 on the token 5, the probability-weighted score is 4 x 0.6 + 5 x 0.4 = 2.4 + 2.0 = 4.4. A second response with 0.9 on 4 and 0.1 on 5 scores 4 x 0.9 + 5 x 0.1 = 3.6 + 0.5 = 4.1. Both would have been 4 under top-token reading, and they differ by 4.4 - 4.1 = 0.3 on a 5-point scale — a 0.3 / 4 = 7.5% difference in the usable range that discrete reading discards entirely. Across 200 evaluations that resolution is the difference between distinguishing two systems and reporting them as tied, because the discrete scale's granularity is 1.0 while the differences being measured are frequently a few tenths. The cost is that this requires token probabilities, so it is unavailable through APIs that do not expose them.
Commonly confused with
A judge-based score is a measurement instrument with its own error, not a ground truth, and the framework's rubric generation improves its consistency without making it correct. Two specific biases survive: a judge tends to prefer outputs from its own model family, which confounds exactly the cross-provider comparisons such scores are used for, and it tends to prefer longer responses regardless of quality. So the comparison a judge can support is between two systems scored by the same judge in the same run, and the one it cannot support is a number quoted across papers, judges or versions.
When to use it
Reach for it when:
- Open-ended generation — summarisation, dialogue, explanation — where no reference answer exists and overlap metrics do not apply.
- Relative comparison within one evaluation run, where the judge's bias is a constant rather than a confound.
- With token probabilities available, since the probability-weighted score is where most of the resolution comes from.
- Reported alongside the judge's measured agreement with human raters on the same task, which bounds the result.
Reach for something else when:
- Comparing systems built on the judge's own model family against others, where self-preference is a direct confound.
- As an absolute quality score, since the number is a property of the judge as much as of the output.
- Where a programmatic check exists — format validity, arithmetic, code that must run — which is cheaper and exact.
- Across different judges or judge versions, where the scale is not shared and the figures are not comparable.
Primary source
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to G-Eval — AI Glossary.
Frequently asked questions
What is G-Eval?
An LLM-as-judge framework that generates evaluation steps via chain-of-thought before scoring.
What concepts are related to G-Eval?
Closely related concepts include llm as judge, eval, chain of thought.