Context Management — AI Glossary

Reference details

Related terms

• Conversation History
• Context Window
• Context Window Management

How it works

1. Before each call the candidate content is tokenised and counted, since a character or word estimate is wrong by enough to matter near the limit.
2. Space is reserved for the response and for anything that must not be dropped — the system block, tool schemas — and the remainder is the budget actually available to history.
3. A policy then selects what fills it: most recent turns, turns scored for relevance to the current query, or chunks retrieved from a store of past turns.
4. Content that does not fit is either dropped or compressed, and compression means a model call that rewrites several turns into fewer tokens, which costs latency on the request that triggers it.
5. The message list is rebuilt from that selection on every call rather than mutated in place, because the model has no state between calls to mutate.
6. The assembled list is counted again before sending, as the compression step's own output length is not known until it has been produced.

Example: Every strategy loses something, and the budget says when

A sliding window drops the oldest turns, so the model forgets what was agreed early. Summarisation compresses them and loses the exact wording a later turn depends on. Importance scoring keeps what it judges relevant and is wrong sometimes. There is no lossless option — the decision is which loss you can tolerate. The budget tells you when the decision arrives. With a 128,000-token window, 2,000 reserved for the system block and 4,000 held back for the response, history has 128,000 - 2,000 - 4,000 = 122,000 tokens available. At roughly 400 tokens a turn that is 122,000 / 400 = 305 turns before anything has to be dropped at all. Summarising older turns at about ten to one extends the same budget to the order of three thousand turns, at the cost of one extra model call each time the summariser runs — which is why the machinery is unnecessary below a few hundred turns and unavoidable above a few thousand.

Commonly confused with

Context management is not memory. It decides what to re-send within the window on each call; memory implies persistent state the model can retrieve later. Systems that appear to remember across sessions are doing retrieval, not remembering. It is also distinct from prompt caching, which reuses computation for a prefix that has not changed — caching makes resending history cheaper without making it smaller, so the two compose rather than substituting for each other.

When to use it

Reach for it when:

• Conversations or agent runs outlive the context window
• Cost scales with history and history keeps growing
• Some early context is genuinely load-bearing and must survive
• Above a few hundred turns, where the 305-turn headroom above has been consumed

Reach for something else when:

• Short interactions that comfortably fit — the machinery adds failure modes
• Summarising content you will later need verbatim
• Trimming before measuring which turns actually matter
• As a substitute for prompt caching, which reduces the cost of history rather than its size

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Context Management — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path