Causal Mask — AI Glossary

Reference details

Related terms

• Decoder-Only Architecture
• KV Cache
• Attention Mechanism

How it works

1. Attention scores are computed for every pair of positions, then entries corresponding to future positions are set to negative infinity before the softmax, so they receive exactly zero weight.
2. The mask is lower-triangular: position i may attend to positions 0 through i and no further, which is what makes the model's prediction at each position depend only on what precedes it.
3. That constraint is what licenses teacher forcing during training. Because no position can see ahead, the loss for every next-token prediction in a sequence is computed in one forward pass over the whole sequence rather than one pass per position.
4. The same property makes the key-value cache correct at inference: a position's keys and values never change as more tokens arrive, so they can be computed once and reused.
5. Removing the mask gives bidirectional attention, which produces better representations and no ability to generate — the architectural fork between encoders and decoders is this one matrix.
6. Variants modify which past positions are visible rather than whether the future is: a sliding window restricts how far back attention reaches, while keeping the future hidden.

Example: What the mask buys in training throughput

Take a 2,048-token training sequence. With the mask, one forward pass computes the model's prediction at every position simultaneously, and each of those predictions is a valid next-token prediction conditioned only on what preceded it — so one pass yields 2,048 training signals. Without the mask you could not do this: to train the prediction at position 500 you would have to feed only the first 500 tokens, then re-run with 501 to train the next, and so on, giving 2,048 forward passes for the same 2,048 signals. That is a factor of 2,048 in training throughput from a triangular matrix of zeros, and it is the reason transformer pretraining is economically possible at all. The same structure pays again at inference in a different currency: because a position's keys and values cannot change when later tokens arrive, they are cached, so generating token n costs one position's worth of work instead of recomputing all n. Both benefits come from the same fact — nothing looks forward — which is why the mask is better understood as the architecture's central design decision than as an implementation detail.

Commonly confused with

The causal mask and a padding mask are both masks and solve unrelated problems. A causal mask hides future positions and is the same triangular pattern for every sequence. A padding mask hides positions that contain filler added to make a batch rectangular, so it differs per sequence and depends on actual lengths. Both are applied before the softmax and a bug in either produces quiet quality loss rather than an error: a missing causal mask lets the model see the answer during training and produces a model that cannot generate, while a missing padding mask lets it attend to filler.

When to use it

Reach for it when:

• Any decoder-only or autoregressive architecture, where it is the mechanism rather than an option.
• Reasoning about why the key-value cache is valid, which follows directly from positions never seeing ahead.
• Understanding why pretraining throughput is what it is, since the mask is what makes teacher forcing possible.
• Diagnosing a model that trains well and generates badly, where mask handling is a standard suspect.

Reach for something else when:

• For representation tasks — classification, retrieval embedding, token labelling — where bidirectional attention is strictly better.
• Confused with a padding mask, which is per-sequence and solves a batching problem.
• Assumed to restrict how far back attention reaches, which is a separate windowing decision.
• As something to tune. It is a structural property; the variants change what of the past is visible, not whether the future is.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Causal Mask — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path