Encoder-Only Architecture — AI Glossary

Reference details

Related terms

• Encoder-Decoder Architecture
• Embedding Model
• Bi-Encoder

How it works

1. The full input is visible at every position because attention is bidirectional — no causal mask. A token's representation is conditioned on everything before and after it, which is the property a generative decoder cannot have.
2. Pretraining uses a masked objective: tokens are hidden and the model predicts them from both directions. That objective is what makes the representations good and also why the model cannot generate fluent continuations.
3. The output is one contextual vector per input token. There is no next-token head, so the model produces representations and a task-specific head turns them into predictions.
4. Sequence-level tasks pool those vectors into one — a pooled token's representation, or a mean — and classify it. Token-level tasks like entity recognition classify each position directly.
5. For retrieval the model is used in one of two shapes: a bi-encoder embeds query and document separately so documents can be indexed offline, or a cross-encoder reads the pair jointly for much better accuracy and no precomputation.
6. Inference is a single forward pass over the whole sequence rather than one pass per generated token, which is why these models are orders of magnitude cheaper than generative ones for classification and scoring.

Example: Why classification does not need a generative model

Classify 1,000,000 documents of about 300 tokens each. A 110M-parameter encoder processes each in one forward pass: roughly 2 x 110e6 x 300 = 6.6e10 FLOPs per document, and 1e6 documents gives about 6.6e16 FLOPs. A 70B generative model asked the same question must process the 300 input tokens and then generate a short answer; the prefill alone is about 2 x 70e9 x 300 = 4.2e13 FLOPs per document, so 1e6 documents is 4.2e19 — a factor of 4.2e19 / 6.6e16 = 636. The encoder is roughly 600 times cheaper for a task where its bidirectional representation is also better suited, since the label depends on the whole text rather than on a left-to-right continuation. The trade is that the encoder needs labelled data to fine-tune its head, while the generative model will do it zero-shot. So the honest decision rule is volume: at a thousand documents the labelling cost dominates and zero-shot wins; at a million the compute ratio dominates and labelling a few thousand examples is the cheaper path by a wide margin.

Commonly confused with

Encoder-only and decoder-only are not generations of the same idea, and treating the first as obsolete is a costly error. A decoder is causally masked so it can generate; an encoder is bidirectional so it can represent. For classification, retrieval scoring and token labelling, bidirectionality is an advantage and generation is unnecessary overhead — which is why encoders still back most production embedding and reranking, where they are both more accurate per parameter and far cheaper to run. The separate confusion is with encoder-decoder models, which have both stacks and do generate.

When to use it

Reach for it when:

• Text classification, entity recognition and token labelling at volume, where the cost difference against a generative model is orders of magnitude.
• Embedding generation for retrieval, where bidirectional representations and offline indexing are what the task needs.
• Reranking as a cross-encoder, which is the most accurate scoring available for a candidate set.
• Latency-constrained paths, where one forward pass beats autoregressive generation.

Reach for something else when:

• Any task requiring generated text — the masked objective does not produce a fluent generator.
• Zero-shot work on a new label set with no labelled data, where a generative model's instruction following wins until volume justifies labelling.
• Long documents without a strategy for the context limit, which is typically far shorter than a modern generative model's.
• Open-ended reasoning, which needs the sequential generation an encoder does not do.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Encoder-Only Architecture — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path