In-Context Learning (ICL) — AI Glossary

Reference details

Related terms

• Few-Shot Prompting
• Zero-Shot Prompting
• Emergent Capabilities
• Context Window

How it works

1. Demonstrations are placed in the prompt and the model infers the mapping they exemplify, then applies it to the new input — with no gradient step and no change to any weight.
2. The mechanism operates entirely within the forward pass: the examples condition the activations, and the inferred task exists only for the duration of that request.
3. Format carries more of the signal than content. Consistent structure across the examples is what makes the pattern legible, and inconsistent formatting degrades the result more than a wrong label does.
4. Example selection and ordering both move performance measurably, with examples nearest the input generally weighted most, which is why retrieving relevant demonstrations per query beats a fixed set.
5. The capability appears with scale rather than being designed in, which is why it is described as emergent and why small models show it weakly.
6. The cost is paid on every request, since the demonstrations are input tokens re-sent each time — which is the whole economic difference from fine-tuning.

Example: The volume at which fine-tuning becomes cheaper

Take 20 demonstrations averaging 150 tokens each: 20 x 150 = 3,000 input tokens appended to every request. At 3 units per million input tokens that is 3,000 x 3 / 1,000,000 = 0.009 units per request. Now suppose a fine-tune that encodes the same behaviour costs 500 units once and removes the demonstrations entirely. The break-even is 500 / 0.009 = 55,556 requests — below that, prompting is cheaper; above it, the fine-tune is. At 1,000,000 requests the comparison is 1,000,000 x 0.009 = 9,000 units of demonstrations against 500 units of training, an 18x difference, and the demonstrations also consume 3,000 tokens of context on every request that could have held something else. The arithmetic alone therefore says fine-tune above roughly fifty thousand requests, and the arithmetic alone is incomplete: the fine-tune is an artefact to version, evaluate and retrain whenever the base model changes, while a prompt change is a deploy. The honest rule is that prompting wins while the task definition is still moving regardless of volume, and the 55,556 figure becomes the deciding factor only once it has stopped.

Commonly confused with

In-context learning is the mechanism and few-shot prompting is the technique that uses it, which is why the terms are used interchangeably and should not be. The distinction matters because the mechanism also operates without examples: a clear task description exploits the same conditioning, which is why zero-shot instruction following works at all. It is also not learning in the sense that fine-tuning is — nothing persists. The model has not acquired the task; it inferred it for one request and will infer it again, at full token cost, on the next.

When to use it

Reach for it when:

• Tasks easier to demonstrate than to describe, where a few examples communicate what a paragraph of instruction cannot.
• Rapid iteration, where the task definition is still changing and each change should cost a deploy rather than a training run.
• Low to moderate volume, where the per-request token cost stays below the amortised cost of a fine-tune.
• With demonstrations retrieved per query rather than fixed, since relevance to the input measurably improves the result.

Reach for something else when:

• High-volume production of a stable task, where per-request demonstration tokens dominate and a fine-tune amortises.
• Where the demonstrations crowd out context the task needs, which is a capacity problem a fine-tune removes.
• With inconsistently formatted examples, which degrade the result more than most people expect.
• Expecting it to persist. Nothing is retained between requests, so every request pays the full cost again.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to In-Context Learning (ICL) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path