Long-Context Model — AI Glossary

Reference details

Related terms

• Context Window
• Sliding Window Attention
• RoPE (Rotary Positional Embedding)
• KV Cache

How it works

1. A model's usable length is set by the positional scheme it was trained with, because attention itself has no notion of distance and position arrives entirely through that encoding.
2. Rotary encodings represent position as a rotation applied to queries and keys, which makes position continuous and therefore extensible by rescaling rather than by retraining from scratch.
3. Extension is usually staged: the base model is trained at a short length, then fine-tuned at progressively longer ones, because a model handed positions it has never seen degrades rather than generalising.
4. The attention cost over the prompt grows with the square of its length, so the engineering that makes long windows affordable targets that term — sliding windows restrict which positions attend to which, and tiled kernels avoid materialising the full score matrix.
5. Those two are different in kind: the kernel changes how the same computation is performed, while a sliding window changes what is computed and so changes what the model can see.
6. None of it makes attention uniform across the window, which is why the length a model accepts and the length over which it reliably attends are measured separately and routinely differ.

Example: The cost of just pasting everything in

A 200-page document is roughly 100K tokens. At $3 per million input tokens that is about $0.30 per question, every question, before any answer is generated. Retrieval over the same document sends perhaps 4K tokens for around $0.012. Long context is not free storage — you re-pay for the whole document on every single call.

100,000 tokens x $3 / 1,000,000 = $0.30 per question
  4,000 tokens x $3 / 1,000,000 = $0.012 per question  # ~25x cheaper

Commonly confused with

A large context window is not the same as reliable use of it. Advertised limits describe what the model accepts, not what it attends to evenly — recall commonly sags for material in the middle. A million-token window does not promise a million tokens of usable attention.

When to use it

Reach for it when:

• The task genuinely needs the whole document at once — global summarisation, cross-references
• Chunking would sever relationships the answer depends on
• The corpus is small and stable enough that re-sending it is affordable

Reach for something else when:

• The same corpus is queried repeatedly; you pay for it every time
• Only a few passages are relevant — retrieval is cheaper and usually more accurate
• The critical detail would land mid-context, where recall is weakest

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Long-Context Model — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path