ModelRefs / FlashAttention — AI Glossary

FlashAttention — AI Glossary

An IO-aware exact-attention algorithm that tiles attention computation to fit in GPU SRAM, eliminating the quadratic memory traffic of standard attention.

Overview

FlashAttention-2 and FlashAttention-3 are the default attention kernels in vLLM, TGI, and PyTorch SDPA. Achieves 2–10× speedup on long contexts at zero accuracy cost by minimizing reads/writes to slow HBM memory.

Reference details

Topic	inference
Last reviewed	2026-06-24

How it works

Standard attention materialises the full score matrix in high-bandwidth memory, writing and re-reading something quadratic in sequence length.
The kernel instead processes queries and keys in tiles small enough to stay in fast on-chip memory.
A running softmax is maintained across tiles with online rescaling, so the normalisation is correct without ever holding the whole matrix.
The backward pass recomputes what it needs from the tiles rather than storing them, trading arithmetic for memory traffic.
The result is mathematically identical to standard attention — the speedup is in memory movement, not in any approximation.

Example: The quadratic term that never reaches memory

At a sequence length of 8,192 the attention matrix is 8,192² ≈ 67 million entries per head; in 16-bit that is roughly 134MB for one head, and a model with 32 heads would move about 4.3GB per layer just writing and re-reading scores. Tiled, those intermediates never leave on-chip memory. Double the sequence to 16,384 and the matrix quadruples to about 537MB per head — which is why the technique's benefit grows with context rather than being a flat percentage.

Commonly confused with

The counterpart to PagedAttention: this one changes the computation, tiling it to keep intermediate results in fast on-chip memory instead of writing the full attention matrix out. It is exact, not an approximation — the outputs are the same as standard attention, which distinguishes it from sparse or windowed variants that deliberately drop pairs to save work.

When to use it

Reach for it when:

Long contexts, where the quadratic intermediate dominates memory traffic.
Explaining a speedup that costs no accuracy, unlike approximate attention.
Training as well as inference, since the recomputation trick targets the backward pass.

Reach for something else when:

Describing KV-cache capacity or fragmentation, which is a different problem.
Very short sequences, where the intermediate is small and the gain marginal.
As an approximation — it is exact, and conflating it with sparse attention misstates the trade.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to FlashAttention — AI Glossary.

Frequently asked questions

What is FlashAttention?