PagedAttention — AI Glossary

Reference details

Related terms

• KV Cache
• vLLM
• Throughput

How it works

1. The KV cache is allocated in fixed-size blocks rather than one contiguous reservation per sequence, with a table mapping a sequence's logical positions to physical blocks.
2. Because blocks need not be adjacent, a sequence grows by acquiring another block instead of needing a large contiguous region reserved up front for its maximum length.
3. That removes the fragmentation and the over-reservation that previously limited how many sequences fit, which is where the throughput comes from.
4. Blocks are reference-counted, so sequences sharing a prefix — the same system prompt, or beam-search branches — point at the same physical blocks instead of duplicating them.
5. Copy-on-write applies when a shared sequence diverges: only the block being written is copied, and the common prefix stays shared.

Example: Where the wasted memory actually was

Reserve for a 4,096-token maximum per sequence when the average request uses 600. Each sequence holds 4,096 tokens' worth of cache and uses about 15% of it, so roughly 85% is reserved and idle. Paged in blocks of 16, a 600-token sequence takes ceil(600/16) = 38 blocks and nothing beyond that. The same memory then holds several times as many concurrent sequences — which is why the gain shows up as throughput, with no change to the attention mathematics at all.

Commonly confused with

Easily confused with FlashAttention because both are named after attention and both raise throughput, but they fix different problems. PagedAttention manages KV-cache *memory*, allocating it in pages so it does not fragment and requests can share prefixes. FlashAttention changes how attention is *computed*. They are complementary and serving stacks generally use both at once.

When to use it

Reach for it when:

• Serving concurrent requests where KV-cache capacity, not compute, caps how many fit.
• Many requests share a long prefix such as a system prompt or few-shot block.
• Explaining why a modern engine serves far more concurrency on the same hardware.

Reach for something else when:

• Single-request latency is the concern — this changes memory management, not attention speed.
• A batch size of one, where there is no fragmentation to recover.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to PagedAttention — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path