ModelRefs / Transfer Learning — AI Glossary

Transfer Learning — AI Glossary

Leveraging a model pretrained on a broad distribution as initialization for a narrower downstream task, amortizing pretraining cost.

Overview

All instruction-tuned and fine-tuned LLMs rely on transfer learning: a base model trained on trillions of tokens provides rich representations that fine-tuning on thousands of examples exploits. Transfer enables sample-efficient task adaptation; the degree of task similarity to pretraining determines transfer quality.

Reference details

Topic	training
Last reviewed	2026-06-24

How it works

A model is pretrained on a broad distribution with a self-supervised objective, which produces representations encoding syntax, semantics and world structure without any task labels.
Those weights become the initialisation for a downstream task, replacing random initialisation. The whole method rests on the claim that a point near a good general solution is closer to a good specific solution than a random point is.
An adaptation objective is applied with a much smaller labelled set and a much lower learning rate, because the pretrained weights are the asset and a large step destroys them.
What is updated is a choice: all parameters, only the final layers, or a small set of inserted parameters as in adapter and low-rank methods. The choice trades adaptation capacity against the amount of pretrained behaviour at risk.
Transfer quality tracks the distance between the pretraining distribution and the target. Close domains transfer with hundreds of examples; distant ones may transfer negatively, where the initialisation is worse than random because the model must first unlearn.
The characteristic failure is forgetting. Updating shared weights for a narrow task degrades the general capability that made the model worth starting from, which is why replay mixtures and parameter-efficient methods exist.

Example: Why the pretraining cost is the number that matters

Take a base model whose pretraining consumed on the order of 10^24 FLOPs, and a fine-tune on 5,000 labelled examples consuming on the order of 10^19. The adaptation is then 10^19 / 10^24 = 10^-5 of the pretraining cost — one hundred-thousandth. That ratio is the entire economic argument for transfer learning: the expensive part is done once by someone else and amortised across every downstream task. It also explains the asymmetry in how the two are treated. A fine-tuning run that goes wrong costs a rerun; a learning rate large enough to destroy the pretrained representations costs an asset you cannot rebuild, since rebuilding it means the 10^24. That is why fine-tuning learning rates sit one to two orders of magnitude below pretraining rates, and why the first check on a disappointing fine-tune is whether the base model's general capability survived it.

Commonly confused with

Transfer learning is the principle; fine-tuning is one implementation of it, and in-context learning is a different mechanism entirely. Fine-tuning changes weights and the adaptation persists. In-context learning changes nothing — examples in the prompt steer behaviour for that request only — so it is not transfer learning despite producing a similar effect on a single task. The practical consequence: in-context learning costs tokens on every request and adapts instantly, fine-tuning costs a training run once and then costs nothing per request. Treating them as interchangeable leads to fine-tuning for a problem a few prompt examples would have solved.

When to use it

Reach for it when:

Labelled data for the target task is in the hundreds or low thousands, where training from scratch is not viable at all.
The target domain is adjacent to the pretraining distribution, so representations are already partly correct.
A consistent behaviour is needed across many requests and paying for examples in every prompt is the dominant cost.
Compute for pretraining is unavailable, which for language models is nearly everyone.

Reach for something else when:

The target distribution is far from pretraining and negative transfer is plausible — the model must unlearn before it can learn.
A few prompt examples already solve it. Fine-tuning adds an artefact to version, evaluate and retrain on every base-model change.
General capability must be preserved and no replay or parameter-efficient strategy is in place, since forgetting is the default outcome.
The task definition is still moving. Each change costs a training run, while a prompt change costs a deploy.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Transfer Learning — AI Glossary.

Frequently asked questions

What is Transfer Learning?