ModelRefs / KV Cache — AI Glossary

KV Cache — AI Glossary

Cached key/value tensors from all previous tokens that let the transformer generate each new token without recomputing earlier positions.

Overview

KV cache is the dominant GPU memory cost at long contexts — it grows linearly with sequence length. PagedAttention, prefix caching, and quantized KV cache all attack this. Multi-query attention (MQA) and grouped-query attention (GQA) reduce KV cache size by sharing heads.

Reference details

Topic	inference
Last reviewed	2026-06-24

How it works

Each attention layer projects every token into a key and a value, and those two tensors are what the cache retains — the query is recomputed per step and never stored.
The size is therefore fully determined by the architecture: two tensors, times the layer count, times the number of key-value heads, times the head dimension, times the bytes per element, per token.
For a 32-layer model with 8 key-value heads and a head dimension of 128 at two bytes per element, that is 2 x 32 x 8 x 128 x 2 = 131,072 bytes per token, which is 128 KiB.
At 8,192 tokens one sequence therefore holds 131,072 x 8,192 = 1,073,741,824 bytes, exactly 1 GiB — and that is per concurrent request, not per model.
Grouped-query attention attacks the key-value head count specifically: sharing one key-value head across four query heads divides the cache by four while leaving the query projections untouched.
Because the term is linear in tokens and the cache is per request, concurrency at long context is bounded by memory rather than by compute — which is why the cache, not the weights, usually decides how many users a card can serve.

Example: Why long contexts get expensive fast

Without the cache, generating token 5,000 means recomputing attention over the preceding 4,999 — generation becomes quadratic and unusable. The cache stores those keys and values so each new token attends to stored state. The cost moves from compute to memory, and that memory grows linearly with context, which is why long-context serving is a memory problem before it is a speed problem.

Commonly confused with

The KV cache is not prompt caching. The KV cache lives for a single sequence and holds per-token tensors in GPU memory. Prompt caching is a provider feature that reuses a shared prefix across separate requests, usually for billing and latency, and is a different mechanism at a different layer.

When to use it

Reach for it when:

It is on by default in every serious serving stack — the decision is how to size it
Reason about it when planning long-context or high-concurrency capacity
Consider quantized KV or GQA when it dominates your memory budget

Reach for something else when:

Do not treat it as free: at long context it can exceed the weights themselves
Do not assume concurrency scales with GPU memory — cache per request bounds it

Referenced by

This term is used by the following ModelRefs references:

become llm engineer

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to KV Cache — AI Glossary.

Frequently asked questions

What is KV Cache?