Attention Mechanism — AI Glossary

Reference details

Related terms

• Transformer
• KV Cache
• FlashAttention

How it works

1. Each token's representation is projected by three learned matrices into a query, a key and a value vector.
2. Every query is dotted with every key, producing a score matrix whose entry at row i and column j is how much position i should draw from position j.
3. Scores are divided by the square root of the key dimension before the softmax, because raw dot products grow with dimension and would otherwise saturate the softmax into a near-one-hot distribution.
4. In a causal model the upper triangle is set to negative infinity before the softmax, so future positions receive exactly zero weight rather than a small one.
5. The softmaxed rows are weights that sum to one, and each output is that weighted average of the value vectors — so attention is a learned, content-dependent averaging rather than a lookup.
6. Multi-head attention runs the whole operation in parallel over several lower-dimensional projections and concatenates the results, which lets different heads attend on different relations within the same layer.

Example: Why doubling context quadruples the cost

Every token attends to every other, so the attention matrix has n² entries. At 1,000 tokens that is 1,000,000 pairs; at 2,000 it is 4,000,000. Double the input, quadruple the attention work. That single fact drives most long-context engineering: sliding windows, grouped-query attention and FlashAttention all exist to bend that curve.

n =  1,000 -> n^2 =     1,000,000 pairs
n =  2,000 -> n^2 =     4,000,000 pairs   # 2x input, 4x work
n = 32,000 -> n^2 = 1,024,000,000 pairs

Commonly confused with

Attention weights are not explanations. A high weight between two tokens shows where the computation looked, not why the output is what it is — attention maps have repeatedly been shown to be unreliable as interpretability evidence.

When to use it

Reach for it when:

• Reasoning about why long context is expensive rather than merely limited
• Choosing between attention variants when memory is the constraint
• Understanding why prefill cost scales worse than generation cost

Reach for something else when:

• As an explanation of model behaviour for an audit or a user
• Assuming all modern models use full quadratic attention — many do not

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Attention Mechanism — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path