ModelRefs / Model Soup — AI Glossary

Model Soup — AI Glossary

Weight-averaging multiple fine-tuned checkpoints of the same base model to improve accuracy and reduce variance.

Overview

Wortsman et al. (2022) showed that averaging weights of models fine-tuned with different hyperparameters (a 'model soup') outperforms any individual soup ingredient. Uniform soup averages all; greedy soup greedily adds models that improve held-out accuracy. Low-overhead ensemble method requiring only weight averaging.

Reference details

Topic	training
Last reviewed	2026-06-24

How it works

Several models are fine-tuned from the same base with different hyperparameters, so they land in nearby regions of the same loss basin.
Their weights are averaged element-wise, which only works because they share an initialisation — averaging unrelated models produces noise.
A uniform soup averages every candidate; a greedy soup adds candidates one at a time and keeps each only if held-out performance improves.
The result frequently beats every ingredient, because averaging cancels idiosyncratic errors while reinforcing what the runs agree on.
Inference cost is unchanged — the output is one model of the original size, which is what distinguishes it from an ensemble.

Example: Why it is free at inference and an ensemble is not

Five fine-tunes ensembled means five forward passes per request: five times the compute and five times the memory to hold them. Averaged into a soup, the result is one model of identical size and one forward pass — the same inference cost as any single ingredient. The accuracy gain is usually smaller than a true ensemble's, and the cost difference is a factor of five, which is why soups are attractive in production and ensembles mostly are not.

Commonly confused with

Easily mistaken for an ensemble, and the distinction is where the combination happens. An ensemble combines predictions at inference, costing one forward pass per member. This combines weights before inference, costing nothing extra and requiring that the members share an initialisation — two independently trained models cannot be averaged usefully. It is also not distillation, which trains a new model to imitate others; nothing is trained here, the arithmetic is a mean over existing checkpoints.

When to use it

Reach for it when:

Several fine-tunes exist from one base and the best single one is being chosen.
Accuracy should improve without raising inference cost.
A held-out set is available, so a greedy soup can be selected rather than guessed.

Reach for something else when:

Candidates come from different base models or initialisations, where averaging destroys them.
Maximum accuracy matters more than inference cost, where an ensemble is stronger.
No held-out set exists to decide which ingredients help.

Primary source

Wortsman et al., Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Model Soup — AI Glossary.

Frequently asked questions

What is Model Soup?