Semantic Cache — AI Glossary

Reference details

Related terms

• Embedding
• Model Gateway (LLM Gateway)
• Inference Cost

How it works

1. An incoming prompt is embedded and compared against the embeddings of previously cached prompts.
2. If the nearest cached entry is within a similarity threshold, its stored response is returned and the model is never called.
3. The threshold is the entire risk control: too loose and semantically different questions receive each other's answers, too tight and almost nothing hits.
4. Entries need invalidation, because a cached answer about changing data becomes wrong while remaining a perfectly good semantic match.
5. Unlike prompt caching, which reuses computation for an identical prefix, this substitutes a different question's answer — so a false hit is a correctness failure, not a performance one.

Example: Where the threshold risk actually bites

At a 0.95 similarity threshold, 'how do I cancel my subscription' and 'how do I pause my subscription' may well exceed it, since the embeddings differ in one word out of six. The cache then answers a pause question with cancellation instructions — a confidently wrong answer that no monitoring flags, because the request succeeded quickly. A 40% hit rate saving 40% of model spend is only a win if the false-hit rate is low enough that the wrong answers cost less than the saving.

Commonly confused with

Frequently confused with prompt caching, and the difference is consequential. Prompt caching reuses computation for an identical prefix and cannot change what the model says. A semantic cache returns a previously stored *answer* because a new question is judged similar — so a similarity threshold that is slightly too loose will confidently serve the answer to a different question. It needs its own correctness evaluation; prompt caching does not.

When to use it

Reach for it when:

• High repetition in incoming questions, such as support or FAQ traffic.
• Answers are stable, so a cached response stays correct over the entry's lifetime.
• The threshold has been tuned against real traffic and false hits measured, not assumed.

Reach for something else when:

• Answers depend on user-specific or fast-changing data, where a correct match returns a wrong answer.
• Prompt diversity is high, giving a hit rate too low to repay the infrastructure.
• You want identical-prefix computation reuse, which is prompt caching and carries no correctness risk.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Semantic Cache — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path