Named Entity Recognition (NER) — AI Glossary

Reference details

Related terms

• Information Extraction
• Text Classification
• PII Detection

How it works

1. Text is tokenised, and the task is framed as a label per token rather than per document, because entities are spans and a span is a contiguous run of tokens.
2. Spans are encoded with a tagging scheme — most commonly BIO, where a token is Beginning, Inside or Outside an entity of a given type — so that two adjacent entities of the same type remain distinguishable.
3. An encoder produces contextual representations and a classification head assigns a tag distribution per token. Context is what resolves 'Washington' as a person, a place or an organisation.
4. Decoding reassembles tags into spans and must enforce validity: an Inside tag cannot open a span. Classical systems used a CRF layer to score whole tag sequences; simpler ones repair invalid sequences after the fact.
5. The LLM route skips tagging entirely and prompts for entities as structured output, which removes the labelled-data requirement and introduces two new problems: the model may return entities whose text does not appear in the source, and character offsets must be recovered by matching back.
6. Evaluation is span-level and exact by default — the boundary and the type must both be right — which is why reported F1 drops sharply on nested or discontinuous entities that the scheme cannot represent.

Example: Why span-level scoring punishes boundary errors twice

A document contains 100 gold entities and the system predicts 100. Suppose 85 are exactly right, 10 have the right type but a boundary off by one token, and 5 are spurious. Under exact span matching a boundary error earns no partial credit: each of the 10 counts as a false negative against the gold span and a false positive for the predicted one. True positives are 85, false positives are 10 + 5 = 15, false negatives are 10 + 5 = 15. Precision is 85 / (85 + 15) = 85%, recall is 85 / (85 + 15) = 85%, F1 is 85%. A token-level read of the same output scores far higher. Take an average span of 3 tokens, so 100 gold entities are 300 tokens: the 85 exact spans contribute 85 x 3 = 255 correct tokens and each boundary error still contributes 2, adding 10 x 2 = 20, so 275 of 300 gold entity tokens are right — 91.7% against the 85% span-level F1. That 6.7-point gap is entirely the scoring scheme, not the system, which is why NER numbers are comparable only when the matching criterion is stated, and why a consumer needing exact offsets should not be sold a token-level score.

Commonly confused with

NER is not entity linking, and the gap between them is where most production disappointment lives. NER finds that a span is an organisation; linking decides which organisation, resolving it to an identifier in a knowledge base. A pipeline that recognises 'Apple' as an ORG has not determined whether it is the technology company, the record label, or a fruit vendor, and every downstream aggregation depends on that. NER is also not PII detection: the categories overlap but the objective differs — PII detection optimises recall against regulatory risk and must catch identifiers, such as account numbers, that no NER scheme defines as named entities.

When to use it

Reach for it when:

• Structured fields must be extracted from prose at volume — parties and dates from contracts, drugs and dosages from clinical notes.
• Entities are the index: search, aggregation or alerting keyed on who and what appears in a document.
• As the first stage of a linking or graph-building pipeline, where spans become candidate nodes.
• Zero-shot via an LLM when the type inventory is bespoke and no labelled data exists, accepting the offset-recovery cost.

Reach for something else when:

• When the question is which specific entity. That is linking, and NER alone leaves the ambiguity in place.
• For nested or overlapping entities under a flat BIO scheme, which cannot represent them and will silently drop the inner or outer span.
• As a compliance control for PII without a recall-oriented evaluation, since a model tuned for F1 is not tuned for the asymmetric cost of a miss.
• On a domain far from the training distribution without re-evaluation. General-domain NER degrades sharply on clinical, legal and financial text, and the degradation is in boundaries as much as types.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Named Entity Recognition (NER) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path