Prompt Caching — AI Glossary

Reference details

Related terms

• KV Cache
• Inference Cost
• System Prompt

How it works

1. The server retains the computed key-value state for a prefix and keys it by a hash of the exact token sequence, so the lookup is on tokens rather than on text.
2. Matching runs from position zero forward and stops at the first divergence, so the cache can only ever skip a leading span — a difference early in the prompt invalidates everything after it.
3. On a hit the prefill for the matched span is skipped and computation resumes at the first new token, which is why the saving is in prefill and never in generation.
4. Writing an entry costs more than ordinary input because the state has to be materialised and stored, so a prefix read back zero times is strictly more expensive than not caching it.
5. Entries expire on a short time-to-live measured in minutes, and the clock is usually refreshed by a read, so a steady trickle of traffic keeps an entry alive and a sparse one does not.
6. Nothing about the model's computation changes — the same state is reused rather than recomputed — which is what makes the feature unable to alter the output.

Example: It pays for itself on the second request

Writing a prefix to the cache typically costs more than a normal input token — call it 1.25× — and reading it back costs a small fraction, call it 0.1×. Send a long shared prompt twice: uncached is 1.0 + 1.0 = 2.0, cached is 1.25 + 0.1 = 1.35. The break-even is inside the first repeat, and from there the saving compounds. The catch is the time-to-live: caches expire after minutes, so a low-traffic endpoint can pay the write premium every single time and never reach a read. Before enabling it, check the request interval against the TTL, not just the prompt size.

Commonly confused with

Prompt caching reuses computation for an identical prefix; semantic caching returns a stored *answer* for a similar question. The first is transparent and cannot change what the model says; the second can serve a stale or subtly wrong response and needs its own correctness thinking. Both differ from the KV cache within a single request, which never outlives it.

When to use it

Reach for it when:

• Long system prompts, few-shot blocks or documents repeated across many requests
• Agent loops and RAG, where a large fixed context is resent every step
• Traffic frequent enough that a cached prefix is read before it expires

Reach for something else when:

• Low-traffic endpoints, where the write premium is paid repeatedly for no read
• Prompts that vary early — the cache matches on a prefix, so put the variable parts last
• Assuming it changes output: it saves recomputation, nothing else

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Prompt Caching — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path