Context Window — AI Glossary

Reference details

Related terms

• Token
• Inference
• RAG (Retrieval-Augmented Generation)
• Attention Mechanism

How it works

1. The limit is on the number of positions the attention mechanism can index, which is fixed by the positional encoding scheme and the lengths the model was trained at.
2. Every request is tokenised and its total counted against that limit, with the prompt and the generated output drawing on the same budget rather than separate ones.
3. Exceeding it is a hard failure rather than a degradation: the request is rejected, or the oldest content is truncated, depending on the client.
4. The KV cache grows linearly with the positions in use, so memory per concurrent request rises with how much of the window is occupied — which is why long contexts reduce how many requests a server can hold at once.
5. Attention cost over the prompt grows faster than linearly, so the last tokens of a long window are more expensive to add than the first.
6. Windows are commonly extended past the trained length by rescaling the positional encoding, which works because the scheme is continuous — but the model was never trained at those positions, so the advertised length and the length at which behaviour holds are different numbers.

Example: Input and output share the budget

With a 128K window, a 120K-token prompt leaves roughly 8K for the response — and if you asked for a long report you will get a truncation instead. The window covers input plus output together, which is why long-document tasks fail at the end rather than the beginning.

Commonly confused with

The context window is not memory. Nothing persists between calls: each request re-sends whatever the model should know. A chatbot that appears to remember is being handed its own history every turn, and paying for it every turn.

When to use it

Reach for it when:

• Sizing whether a document fits before building around it
• Budgeting prompt against expected response length
• Deciding between long context and retrieval

Reach for something else when:

• As a proxy for capability — a bigger window says nothing about reasoning quality
• Assuming attention is uniform across it; middle-of-context recall is measurably weaker

Referenced by

This term is used by the following ModelRefs references:

• short term memory

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Context Window — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path