BERTScore — AI Glossary

Reference details

Related terms

• BLEU
• ROUGE
• Embedding Model

How it works

1. Candidate and reference are both encoded by a contextual embedding model, so each token's vector reflects its surroundings rather than the word alone.
2. Every reference token is matched to its most similar candidate token by cosine similarity, and the average of those maxima gives recall.
3. The same greedy matching in the other direction gives precision, and the two combine into an F-measure.
4. Token importance can be weighted by inverse document frequency, so matching a rare content word counts for more than matching an article.
5. Scores are rescaled against a baseline because raw cosine similarities are compressed into a narrow high range and look uniformly good.

Example: The case n-gram overlap scores at zero

Reference 'the cat sat on the mat' against candidate 'the feline rested upon the rug'. Unigram overlap matches only 'the', so a lexical metric scores near 0.33 on recall and treats the paraphrase as a failure. Contextual embeddings place feline near cat, rested near sat and rug near mat, so the greedy matching finds high similarity for each reference token and the score reflects that the meaning survived. That gap is the entire reason the metric exists, and it is why it is preferred for abstractive output.

Commonly confused with

Grouped with n-gram metrics as a drop-in improvement, which understates the difference. Lexical metrics count surface overlap and cannot credit a paraphrase; this compares meaning via embeddings and can. The trade is interpretability and cost: a lexical score decomposes into which n-grams matched, while this produces a number whose derivation requires running a model. It is also not an LLM judge — it measures similarity to a reference and still needs one, where a judge can score an output against criteria with no reference at all.

When to use it

Reach for it when:

• Abstractive output where correct paraphrases must not be penalised.
• A reference exists and automated scoring is needed at volume.
• Lexical metrics are known to under-credit valid rewordings.

Reach for something else when:

• No reference text exists, where a criteria-based judge is the only option.
• Interpretability matters and you need to know which parts matched.
• Factual correctness is the question — high similarity to a reference is not truth.

Primary source

• Zhang et al., BERTScore: Evaluating Text Generation with BERT (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to BERTScore — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path