Prompt Compression — AI Glossary

Reference details

Related terms

• Context Management
• Long-Context Model
• RAG (Retrieval-Augmented Generation)

How it works

1. A smaller model scores the importance of tokens or sentences in the prompt, and the low-scoring ones are removed before the prompt reaches the main model.
2. Token-level pruning exploits the fact that natural language is redundant for a model's purposes: function words and repeated framing carry little task information even though removing them makes the text awkward to read.
3. Sentence- or passage-level selection keeps the text readable and compresses less, which is the usual trade between aggressive and conservative methods.
4. A third approach trains a model to emit a short sequence of continuous summary vectors, which compresses further still and produces something no longer inspectable as text.
5. The instruction and the question are protected from compression, since they are the parts whose exact wording determines what the model does.
6. The compressor's own cost must be counted, and it is what determines whether the saving is real — a compressor approaching the main model's cost per token eliminates the benefit.

Example: Netting the compressor's cost against the saving

Take a 10,000-token prompt compressed 4x to 2,500, saving 7,500 input tokens per request. Across 1,000,000 requests that is 1,000,000 x 7,500 = 7,500,000,000 tokens, and at 3 units per million, 7,500 x 3 = 22,500 units saved. Now the compressor. It must read the full 10,000 tokens on every request, so it processes 1,000,000 x 10,000 = 1e10 tokens; at a small model's 0.1 units per million that is 10,000 x 0.1 = 1,000 units. The net saving is 22,500 - 1,000 = 21,500 units, so the compressor costs 1,000 / 22,500 = 4.4% of what it saves — comfortably worth it. The arithmetic turns at a compressor costing 2.25 units per million, where its 22,500 units would consume the entire saving: that is 2.25 / 3 = 75% of the main model's rate, so the method works as long as the compressor is meaningfully cheaper than the model it feeds. What the arithmetic does not price is the accuracy cost, which published methods put in the low single digits and which has to be measured on your own task rather than assumed.

Commonly confused with

Prompt compression removes tokens from a prompt; prompt caching avoids re-billing tokens that have not changed. The two address the same cost from opposite directions and compose badly in one specific way: compression makes the prompt variable, which can defeat a cache that depends on a stable prefix. So for a workload with a large fixed context, caching is usually the better tool and compression is for prompts that differ every request. The separate confusion is with context selection — choosing which documents to include is a retrieval decision, while compression operates on text already chosen.

When to use it

Reach for it when:

• Long prompts that differ on every request, where caching cannot help and the tokens are paid in full.
• High volume, where the per-request saving multiplies past the compressor's own cost.
• Where the context is demonstrably redundant, which token-level methods exploit and terse context does not offer.
• With the accuracy cost measured on your own task, since the published figures are for other tasks.

Reach for something else when:

• Where prompt caching applies, since a stable prefix is cheaper to cache than to compress and compression can defeat the cache.
• On instructions and questions, whose exact wording determines the model's behaviour.
• With a compressor whose cost approaches the main model's rate, which eliminates the saving.
• On precision-critical content — legal text, numbers, identifiers — where a removed token changes the meaning rather than the style.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Prompt Compression — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path