Model Merging — AI Glossary

Reference details

Related terms

• Model Soup
• LoRA (Low-Rank Adaptation)
• Adapter (PEFT)

How it works

1. Merging requires a shared origin. The models must be fine-tunes of one base checkpoint, so their weights are coordinates in the same space and arithmetic on them is meaningful; merging unrelated models produces noise.
2. The simplest method is weight interpolation — a weighted average, parameter by parameter. Spherical interpolation follows the arc between two weight vectors instead of the chord, which preserves vector magnitude and in practice interpolates better.
3. Task-vector methods first subtract the base weights from each fine-tune, giving a delta that represents what that fine-tune learned, then combine the deltas and add them back.
4. Interference between deltas is the central problem: two fine-tunes often want to move the same parameter in opposite directions. Trimming each delta to its largest-magnitude entries and resolving sign conflicts by majority reduces the cancellation that naive averaging produces.
5. Random dropping with rescaling exploits delta redundancy — discarding most entries and scaling the rest up preserves the task behaviour while reducing overlap between the deltas being combined.
6. The whole operation is weight arithmetic with no gradients, so it runs on CPU in minutes and the real cost is evaluating the result, which is also the only way to know whether it worked.

Example: Why merging is free and finding a good merge is not

Averaging two 7B checkpoints is 7e9 additions and multiplications — seconds of CPU work, no accelerator, no gradients. The search is the cost. A linear merge of two models has one weight to set; sampling it at 0.1 intervals from 0 to 1 is 11 candidates. Three models with weights summing to 1 sampled at the same granularity give 66 combinations, since the count is C(10 + 2, 2) = 66. Add a method choice among four and a trim threshold among three and the grid is 66 x 4 x 3 = 792 candidates. If evaluating one candidate on a suite that actually distinguishes them takes 20 minutes, the full grid is 792 x 20 = 15,840 minutes, about 11 days of continuous evaluation. So the honest description is that merging is computationally free and empirically expensive: the published recipes exist because the search was already run, and reproducing a result is minutes while finding a new one is an evaluation budget. The corollary is that a merge is only worth attempting if you already have an evaluation suite you trust.

Commonly confused with

Merging is not ensembling, and the difference is inference cost. An ensemble keeps every model and runs them all, paying N forward passes per request for the benefit of genuinely independent predictions. A merge produces one model of the original size, so inference costs exactly what one model costs — and it is not averaging predictions but averaging parameters, which is a much stronger assumption and can simply fail. The related confusion is with distillation: distillation trains a student on a teacher's outputs and needs gradients and data, while merging needs neither and only works between relatives.

When to use it

Reach for it when:

• Combining fine-tunes of one base model whose capabilities are complementary — a code tune and an instruction tune.
• Averaging several runs of the same fine-tune with different seeds or hyperparameters, which reliably reduces variance.
• When inference cost rules out an ensemble but you want some of its benefit.
• With an evaluation suite already in place, since the suite is what the method actually costs.

Reach for something else when:

• Across different base models or architectures, where the weight spaces are unrelated and the result is noise.
• Expecting reliable gains from an untested recipe. Merges fail, and the only detector is evaluation.
• When genuinely independent predictions are the requirement, which is an ensemble and cannot be merged into one model.
• As a substitute for fine-tuning on a capability no constituent model has. Merging recombines what exists; it does not add.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Model Merging — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path