Decoder-Only Architecture — AI Glossary

Reference details

Related terms

• Encoder-Decoder Architecture
• Causal Mask
• Transformer

How it works

1. A single stack of transformer blocks is trained on one objective: given every token so far, predict the next one.
2. A causal mask zeroes attention to future positions, so position i can see only positions ≤ i. That is the entire structural difference from a bidirectional encoder.
3. The mask makes the loss parallel over the sequence during training — every position predicts its successor in one pass — which is what makes the architecture cheap to scale.
4. At inference, generation is sequential: emit a token, append it, attend again. Prior keys and values are cached because the mask guarantees they cannot change.
5. Prompting works because there is no architectural boundary between input and output. Instructions, examples and the answer are all just earlier tokens in the same sequence.

Example: Why the cache is sound, and why it is necessary

Generating 500 tokens after a 1,000-token prompt. Without caching, each new token re-attends over the whole prefix: roughly 1,000 + 1,001 + … + 1,499, on the order of 625,000 position-pairs. With caching, the prompt is processed once and each step attends over the cache plus one new entry. Causal masking is what licenses this — a past token's representation cannot depend on a future one, so yesterday's keys are still valid. The same mask that limits what the model sees is what makes serving it affordable.

Commonly confused with

Contrasted with encoder-decoder as though the difference were capability, when it is conditioning. An encoder-decoder encodes the input bidirectionally and attends to it through cross-attention, which suits tasks with a clean input/output split such as translation. A decoder-only model puts everything in one causal stream, which is strictly less expressive about the input — it cannot look rightward within the prompt — and turned out to matter less than the scaling and prompting advantages of a single objective. Encoder-only models like BERT are the third branch and are not generative at all, which is why they persist for embedding and classification.

When to use it

Reach for it when:

• Explaining why one architecture dominates general-purpose language models.
• Explaining why prompting works at all, and why few-shot examples need no architectural support.
• Reasoning about KV-cache behaviour, prefill cost, and why they differ.

Reach for something else when:

• Describing embedding or classification models, which are typically encoder-only and bidirectional.
• As shorthand for 'modern' — plenty of current multimodal and speech systems still use cross-attention over an encoded side.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Decoder-Only Architecture — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path