Late Interaction (ColBERT) — AI Glossary

Reference details

Related terms

• Bi-Encoder
• Dense Retrieval
• Reranking

How it works

1. Documents are encoded offline into one embedding per token rather than one per document, so a 200-token passage stores 200 vectors. The index is therefore far larger than a single-vector index of the same corpus — the cost that buys the quality.
2. The query is encoded at search time, also per token, producing one vector per query token.
3. Scoring is MaxSim: for each query token, find its highest similarity against any document token, then sum those maxima across query tokens. Interaction between query and document happens at scoring time — hence 'late' — rather than being compressed into one vector beforehand.
4. Retrieval is still approximate-nearest-neighbour, but over token vectors: candidate documents are gathered by their individual token matches, then fully scored with MaxSim.
5. The architectural point is where interaction occurs. A bi-encoder interacts never, comparing two fixed vectors. A cross-encoder interacts fully, running query and document jointly through a transformer, which forbids precomputation. Late interaction precomputes the documents and defers only the cheap comparison.
6. Compression is what makes it practical: residual quantisation of token vectors cuts index size substantially, and later variants treat that compression as part of the design rather than an afterthought.

Example: Where the index cost and the latency saving land

A corpus of 1M passages averaging 200 tokens. A single-vector bi-encoder at 768 dimensions in 4-byte floats stores 1e6 x 768 x 4 = 3.07GB. Late interaction at 128 dimensions per token stores 1e6 x 200 x 128 x 4 = 102.4GB — 33x larger, which is why quantisation matters: at 1 byte per dimension it is 25.6GB, and at the 2-bit residual compression the later variants use, closer to 6.4GB. Now the other side. A cross-encoder reranking 100 candidates runs 100 transformer forward passes over query-document pairs at query time; at 8ms each that is 800ms. Late interaction does zero transformer passes over documents at query time — they were encoded offline — and only a MaxSim over token vectors, typically single-digit milliseconds. So the trade is explicit: roughly 2-8x the storage of a bi-encoder after compression, for most of a cross-encoder's quality at roughly a hundredth of its query latency.

Commonly confused with

Late interaction is not reranking, though it is often compared to it, and the two compose rather than compete. A reranker is a stage — take k candidates from a cheap retriever and rescore them, usually with a cross-encoder. Late interaction is a retrieval architecture that scores the whole corpus, so it can serve as the first-stage retriever and remove the need for a separate rerank pass. The confusion has a practical cost: teams add a late-interaction model as a reranker over a bi-encoder's top 50, which keeps the bi-encoder's recall ceiling and pays for the large index without using it.

When to use it

Reach for it when:

• Recall matters more than index size, and the corpus is large enough that a cross-encoder over everything is impossible.
• Queries are long or multi-aspect, where a single query vector averages away the terms that should match.
• Out-of-domain retrieval, where token-level matching degrades more gracefully than a single embedding tuned on another distribution.
• Exact terms matter — identifiers, product codes, names — which single-vector embeddings routinely smear.

Reach for something else when:

• Storage or memory is the binding constraint and compression is not acceptable, since the index is multiples of a single-vector one.
• The corpus is small enough to cross-encode directly, where a cross-encoder gives better quality with no index at all.
• Short keyword queries on a well-matched domain, where a tuned bi-encoder plus a reranker is cheaper and close in quality.
• Bolted on as a reranker over a weak first stage, which pays the index cost while keeping the first stage's recall ceiling.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Late Interaction (ColBERT) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path