BLEU — AI Glossary

Reference details

Related terms

• ROUGE
• BERTScore
• Exact Match (EM)

How it works

1. Precision is computed separately at each n-gram order from one to four, counting how many of the candidate's n-grams appear in the reference.
2. Each count is clipped at the number of times that n-gram occurs in the reference, which stops a candidate from scoring by repeating one matching phrase.
3. The four precisions are combined as a geometric mean rather than an average, and that choice has a consequence: a zero at any single order sends the whole score to zero.
4. A short candidate can achieve high precision by saying very little, so a brevity penalty multiplies the result down when the candidate is shorter than the reference.
5. The metric is defined over a corpus, not a sentence: n-gram counts are summed across all sentences before any division, so one bad sentence cannot zero the corpus figure.
6. Applying it per sentence breaks that protection, since a single sentence frequently has no four-gram match at all — which is why sentence-level use requires a smoothing rule, and why two implementations can report different numbers for identical text.

Example: Score a correct paraphrase and watch it lose

Candidate “the cat sat on the mat”, reference “the cat is on the mat”. Unigrams: the (clipped to 2), cat, on and mat match; sat does not — 5 of 6, or 0.83. Bigrams: “the cat”, “on the” and “the mat” match; “cat sat” and “sat on” do not — 3 of 5, or 0.60. Both are six tokens, so the brevity penalty is 1. Now try a fluent human paraphrase — “the mat had a cat on it” — and bigram precision collapses to zero despite the meaning being intact. BLEU measures surface overlap with the references you supplied, and nothing else.

Commonly confused with

BLEU is precision-oriented; ROUGE is recall-oriented and is the summarisation convention. Neither measures meaning. BERTScore compares embeddings instead, and an LLM-as-judge rubric compares against criteria — both disagree with BLEU on paraphrase, which is usually a point in their favour.

When to use it

Reach for it when:

• Machine translation, where decades of published results make it the comparable metric
• Regression testing a system against its own previous output, where the reference is stable
• Cheap, deterministic scoring in CI, where you need a number that never drifts

Reach for something else when:

• Open-ended generation, chat or summarisation — many correct answers, few references
• Comparing systems that differ in style or verbosity rather than in accuracy
• Any claim about quality made from BLEU alone without reporting the tokenization used

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to BLEU — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path