ModelRefs / Curriculum Learning — AI Glossary

Curriculum Learning — AI Glossary

A training strategy presenting examples in increasing order of difficulty, mimicking human learning to improve convergence and generalization.

Overview

Curriculum learning (Bengio et al. 2009) orders training data by difficulty—simple examples first, hard ones later. For LLMs, curricula may progress from short to long contexts, or from common to rare vocabulary. Used in code generation models (EasyToHard curricula for algorithmic tasks) and math reasoning training.

Reference details

Topic	training
Last reviewed	2026-06-24

How it works

A stage boundary is implemented as a checkpoint handoff: the weights from one stage initialise the next, and everything else about the run is reconfigured.
Optimizer state is usually discarded at the boundary rather than carried over, because the moment estimates were accumulated against a different data distribution.
The learning rate is then re-warmed from near zero, since resuming at the previous rate applies large updates to weights that are already well fitted to the earlier stage.
The data mixture changes at the same point, and that change is the curriculum — the ordering is expressed entirely as which distribution each stage draws from.
A later stage's gradients can erase behaviour the earlier one installed, so the earlier stage's evaluation is re-run after the later one rather than assumed to still hold.
That re-run is the only reliable check, because a loss curve on the current stage's data looks healthy whether or not the previous stage's capability survived.

Example: You are probably already doing it, under another name

Pretraining data is generally shuffled, so the classic easy-to-hard ordering is not what most LLM training does. The curricula that are standard practice go unlabelled: training at a short context and then extending to a long one; supervised fine-tuning before preference optimisation; simple verifiable tasks before harder ones in reasoning training. Each is an ordered progression where the later stage depends on competence built in the earlier one. Recognising them as curricula is useful because it makes the design question explicit — what has to be true before this stage can work, and what happens if the order is reversed.

Commonly confused with

Curriculum learning orders the data; active learning chooses which examples to label next based on uncertainty; continual learning is about adding new tasks without losing old ones. All three concern what the model sees when, and they answer different questions — only the curriculum is decided entirely in advance.

When to use it

Reach for it when:

Context-length extension, where short-then-long is the established path
Reasoning training, where verifiable easy tasks establish a base for harder ones
Multi-stage post-training, where each stage assumes the previous one succeeded

Reach for something else when:

General pretraining, where shuffled data is the default for good reason
As a fix for insufficient or low-quality data — ordering bad data does not improve it
Without checking for forgetting: a late stage can erase what an early one taught

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Curriculum Learning — AI Glossary.

Frequently asked questions

What is Curriculum Learning?