Model Pruning — AI Glossary

Reference details

Related terms

• Quantization
• Model Distillation
• Knowledge Distillation

How it works

1. An importance score is assigned to each candidate for removal — a weight, a row, an attention head, a whole layer. Magnitude is the cheapest proxy; gradient- and Hessian-based scores are better and cost more to compute.
2. Unstructured pruning zeroes individual weights anywhere in the tensor. It achieves the highest compression for a given accuracy loss, because it is free to remove exactly the least useful parameters.
3. Structured pruning removes whole units — heads, channels, layers — so the remaining tensor is smaller and dense. The accuracy cost is higher for the same sparsity, and the result runs faster on ordinary hardware without any special support.
4. Semi-structured patterns sit between the two: a fixed ratio of zeros within each small block, which some accelerators can skip in hardware, giving a sparse speedup with a constrained pattern.
5. Recovery follows removal. A pruned model is fine-tuned, often with distillation from the unpruned version as the teacher, and most of the recoverable accuracy comes back in this step rather than from a better importance score.
6. The decisive question is whether the sparsity translates into anything. Zeros stored in a dense tensor occupy the same memory and cost the same arithmetic, so unstructured sparsity without a sparse format and sparse kernels is a change in the numbers and not in the bill.

Example: Why 90% sparsity is not a 10x saving

A 16-bit dense model stores 2 bytes per parameter. Prune 90% of the weights and the naive expectation is a 10x memory reduction, since only a tenth of the values remain. But a sparse format must store position as well as value: with a 2-byte value and a 2-byte index per surviving weight, the cost is 0.1 x (2 + 2) = 0.4 bytes per original parameter, so the real reduction is 2 / 0.4 = 5x, not 10x. The index overhead consumed half the benefit. Push to 99% sparsity and it is 0.01 x 4 = 0.04 bytes against 2, a 50x reduction — the indexing overhead matters less as density falls, which is why extreme sparsity is where unstructured pruning is worth the kernel complexity. Compare structured pruning that removes 25% of attention heads: the tensors are genuinely smaller and dense, so the saving is exactly 25% with no format change, no index, no special kernel and no risk that the sparsity fails to materialise. A guaranteed 25% frequently beats a theoretical 5x that depends on a sparse kernel being available on the hardware you actually have.

Commonly confused with

Pruning and quantisation reduce size along different axes and compose rather than compete. Quantisation reduces the bits per weight while keeping every weight; pruning removes weights while keeping the precision of those that remain. Combining them multiplies: a 4-bit model with 25% of its heads removed is smaller than either alone. The confusion that costs real effort is between pruning and distillation — distillation trains a smaller model from scratch to imitate a larger one, which gives full freedom over the student's architecture, while pruning is constrained to a subnetwork of the original and therefore inherits its shape.

When to use it

Reach for it when:

• Structured pruning when inference cost on ordinary hardware is the goal, since the saving is real without kernel support.
• Semi-structured patterns where the accelerator supports them in hardware, which is where sparse speedups are actually delivered.
• Alongside quantisation and a distillation-based recovery pass, which is the combination that preserves the most accuracy per byte removed.
• Where the pruned model will be used at volume, so a one-time recovery fine-tune amortises.

Reach for something else when:

• Unstructured pruning without sparse kernels on the target hardware, where zeros cost exactly what weights cost.
• Expecting sparsity to translate linearly into savings — index overhead at moderate sparsity takes a large fraction of it.
• Without a recovery pass. Most of the accuracy lost to removal is recoverable, and skipping that step misattributes the loss to pruning.
• When a smaller model already exists. Pruning a large model to a given size rarely beats a model trained at that size, unless the pruned one inherits capability the smaller one was never trained for.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Model Pruning — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path