Exact Match (EM) — AI Glossary

Reference details

Related terms

• BLEU
• ROUGE
• RAGAS

How it works

1. Both prediction and reference are normalised by a fixed, published procedure: lowercase, strip punctuation, remove articles, collapse whitespace. The normaliser is part of the metric — changing it changes the score, so an EM number without its normaliser is not reproducible.
2. The normalised strings are compared for equality. The result per example is 1 or 0; there is no partial credit, which is the whole design.
3. Where a dataset supplies several acceptable references, the prediction scores 1 if it matches any of them, so reference coverage directly determines the ceiling.
4. The reported score is the mean over examples, which makes it a proportion and gives it a binomial standard error — the basis for deciding whether two EM scores differ at all.
5. Token-overlap F1 is almost always reported alongside, because the gap between EM and F1 localises the failure: a large gap means answers are nearly right, a small gap means they are wrong.
6. Generative models need an answer-extraction step before scoring, since a model that replies in a sentence fails string equality against a bare entity. That extractor is a confound and is frequently the real difference between two reported numbers.

Example: Why a 2-point EM difference is usually nothing

Two models score 71% and 73% EM on a 1,000-question set. The standard error of a proportion is sqrt(p(1-p)/n), so at p = 0.72 and n = 1,000 that is sqrt(0.72 x 0.28 / 1000) = sqrt(0.0002016) = 0.0142, about 1.4 points. The difference of two independent proportions has standard error sqrt(0.0142^2 + 0.0142^2) = 0.0201, about 2.0 points — so a 2-point gap is roughly one standard error, about what sampling alone produces. To resolve a 2-point difference at this accuracy you need roughly 4x the questions, since halving a standard error requires quadrupling n: 4,000 questions brings the difference's standard error to about 1.0 point. Meanwhile a model whose answers are phrased as sentences can lose more than 2 points to the answer extractor alone, which is why leaderboard-adjacent EM gaps of this size should move nothing.

Commonly confused with

Exact match is not accuracy, though it is reported in the same units. Accuracy presumes a closed label set where the correct answer is one of k choices; EM is string equality against a reference in an open output space, so it penalises correct paraphrases, different surface forms of the same entity, and any answer the dataset's reference list omits. It is also not a quality measure for generation: an EM of 0 on a free-text summarisation task says the metric does not apply, not that the output is bad.

When to use it

Reach for it when:

• Short factual answers with a canonical surface form — dates, names, numbers, single entities.
• Extraction and structured-output tasks where the field value must be byte-correct downstream.
• Regression testing, where the strictness is a feature: any drift in surface form is caught.
• Alongside token-level F1, where the EM-to-F1 gap tells you whether errors are near-misses or genuine.

Reach for something else when:

• Free-form generation — summaries, explanations, dialogue — where many surface forms are correct and EM measures conformity to one.
• As the only metric on a QA set with sparse reference lists, since the ceiling is set by reference coverage rather than model capability.
• Comparing scores produced by different normalisers or different answer extractors, which is the most common way EM comparisons mislead.
• On small evaluation sets, where the binomial standard error swamps the differences being discussed.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Exact Match (EM) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path