ALiBi (Attention with Linear Biases) — AI Glossary

Reference details

Related terms

• Positional Encoding
• RoPE (Rotary Positional Embedding)
• Long-Context Model

How it works

1. A bias matrix is added to the attention scores before the softmax, with each entry equal to the negative distance between its query and key positions multiplied by a per-head slope.
2. Slopes are assigned as a geometric sequence across heads — a half, a quarter, an eighth and so on — so the heads span a range of effective windows rather than all decaying at one rate.
3. A head with a steep slope attends almost entirely locally while a shallow-sloped head still reaches distant tokens, and the model has both available in every layer.
4. Nothing here is learned: the slopes are fixed by a formula from the head count, so the scheme adds no parameters and no embedding table at all.
5. Because the bias depends only on the index difference, the matrix for any sequence length can be computed on demand or cached once per length and reused across every layer and request.
6. That is also why running past the trained length does not fail — a larger gap simply produces a larger penalty, which is defined for every distance rather than looked up from positions the model has seen.

Example: A penalty proportional to distance, learned by nobody

Instead of adding position information to the inputs, this subtracts a penalty from the attention score itself, proportional to how far apart two tokens are, with a fixed per-head slope. On a head with slope 1/2, a gap of 100 tokens costs 50 and a gap of 200 costs 100. Attention therefore decays with distance without ever being cut off. Because no position is ever learned, nothing breaks when the sequence runs longer than anything seen in training — the penalty is simply computed for the larger gap. That is the property the method was built for, and it is a different mechanism from rotating the query and key vectors.

Commonly confused with

All three positional schemes are commonly lumped together and behave differently. Learned absolute embeddings add a vector per position at the input and cannot extrapolate past the trained length. Rotary encoding rotates query and key vectors so scores depend on relative offset. ALiBi adds a distance-proportional bias to the scores. Only the last two extrapolate by construction, and modern open-weights models overwhelmingly chose rotation.

When to use it

Reach for it when:

• Reading or reproducing models from the era that adopted it
• Understanding why extrapolation behaviour differs so much between architectures
• Settings where a monotonic recency bias is a desirable inductive bias

Reach for something else when:

• New architectures, where rotation plus a context-extension method is the current default
• Tasks needing strong attention to distant tokens, which the penalty works against
• Assuming extrapolation is unlimited — quality still degrades well beyond training length

Primary source

• Press et al., Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to ALiBi (Attention with Linear Biases) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path