Sliding Window Attention — AI Glossary

Reference details

Related terms

• Attention Mechanism
• Long-Context Model
• Grouped Query Attention (GQA)

How it works

1. Each position attends only to a fixed number of preceding positions rather than to the whole prefix, which replaces the quadratic term with one linear in sequence length and in the window size.
2. Stacking layers extends reach beyond the window: a position sees its window directly, and each of those positions saw their own window in the layer below, so the effective receptive field grows with depth.
3. That growth is the architecture's central claim — local attention plus depth approximates global attention — and its limitation, since information from far away arrives indirectly through intermediate summarisation rather than by direct lookup.
4. The key-value cache stops growing once the window is full, because positions outside it are never attended to and need not be retained, which is the serving benefit.
5. Designs commonly interleave a few full-attention layers among the windowed ones, restoring exact long-range lookup at a fraction of the cost of making every layer global.
6. Some positions may be designated global, attended to by everything, which gives a cheap channel for information that must be available everywhere.

Example: Receptive field against cost, both from the same two numbers

Take a window of 4,096 and 32 layers. The effective receptive field is the window times the depth: 4,096 x 32 = 131,072 positions, so in principle information can travel the length of a 131,072-token sequence. The cost side uses the same numbers. Full attention over 131,072 positions scales with the square of the sequence, while windowed attention scales with sequence times window, so the ratio is 131,072 / 4,096 = 32 — a 32-fold reduction in attention work at that length, and the saving grows linearly as sequences get longer. The cache benefit is more concrete: at 0.5MB per token for a model of this shape, full attention at 131,072 tokens needs 131,072 x 0.5 = 65,536MB, about 65.5GB, while a windowed model retains only the window: 4,096 x 0.5 = 2,048MB, about 2GB, regardless of how long the sequence becomes. That is a factor of 32 again, and it is the difference between a model that fits and one that does not. What the arithmetic does not show is quality: information crossing 32 layers to travel 131,072 positions has been re-summarised 32 times, which is why interleaved global layers exist.

Commonly confused with

Sliding-window attention is not a state-space model, though both achieve linear scaling. A windowed attention model still attends — every position in the window is individually addressable and retrievable exactly — so it loses access to distant tokens while retaining exact access to near ones. A recurrent state compresses the entire past into a fixed summary, so nothing is individually addressable. The failure modes differ accordingly: windowed attention fails on long-range dependencies that the depth-based receptive field summarises away, and a recurrent state fails on exact retrieval at any distance.

When to use it

Reach for it when:

• Long-context serving where the growing key-value cache is the binding constraint, which the window bounds.
• Tasks whose dependencies are predominantly local — most language modelling, code within a function, conversational turns.
• With interleaved global layers when exact long-range lookup is also needed, which is the standard production shape.
• Extending a model trained at shorter context to longer sequences, which the bounded cache makes practical.

Reach for something else when:

• Tasks requiring exact retrieval from far outside the window, where depth-based reach is a summarisation rather than a lookup.
• Without checking where the receptive field actually ends, since window times depth is an upper bound on reach rather than a guarantee of quality at that distance.
• In a purely windowed configuration when long-range dependencies matter, where interleaved global layers are what make it work.
• Assumed equivalent to full attention at the same context length, which the re-summarisation argument says it is not.

Primary source

• Beltagy et al., Longformer: The Long-Document Transformer (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Sliding Window Attention — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path