Reranker (Cross-Encoder) — AI Glossary

Reference details

Related terms

• RAG (Retrieval-Augmented Generation)
• Hybrid Search
• Embedding

How it works

1. The query and one candidate are concatenated into a single sequence with a separator, so attention runs across the pair and every query token can address every passage token.
2. The model emits one relevance value per pair, read from a classification head or from the logit of a designated token, rather than producing an embedding.
3. Because the pair is the unit, each candidate is its own forward pass and the cost is linear in candidate count — which is the whole reason the stage operates on a shortlist.
4. Those passes are independent, so they are batched into one or two GPU calls, and that batching is what makes a hundred candidates a single-digit-millisecond addition rather than a hundred sequential inferences.
5. The resulting scores are comparable only within one query's candidate set, since nothing in training related the score for one query to the score for another.
6. Passage length matters more here than in retrieval: the pair must fit the model's sequence limit, so a long candidate is truncated and is scored on its opening rather than on the part that mattered.

Example: Why a second model earns its latency

A bi-encoder embeds query and document separately, so the two never interact until a dot product at the end — fast enough to scan millions. A cross-encoder feeds the pair through together, letting every query token attend to every document token, which is far more accurate and far too slow for the whole corpus. So you retrieve 100 cheaply, then rerank those 100 properly.

Commonly confused with

A reranker is the model; reranking is the pipeline stage it occupies. A reranker cannot improve recall — it only reorders what retrieval already returned. If the right document was not in the top 100, no reranker will surface it.

When to use it

Reach for it when:

• Retrieval recall is acceptable but the top few results are poorly ordered
• You feed a small number of passages into a context-limited prompt
• Precision at position 1–3 is what actually drives answer quality

Reach for something else when:

• Recall is the problem — fix retrieval first; reranking cannot recover a miss
• Latency budget is tight and the ordering is already good enough
• The candidate set is already small enough to pass through whole

Referenced by

This term is used by the following ModelRefs references:

• long term memory rag
• vector search tool
• naive rag stack
• advanced rag stack

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Reranker (Cross-Encoder) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path