Parameter-Efficient Fine-Tuning (PEFT) — AI Glossary

Reference details

Related terms

• LoRA (Low-Rank Adaptation)
• Fine-Tuning

How it works

1. Base weights are frozen, so the pretrained behaviour is preserved and only a small set of new parameters receives gradients.
2. Where those parameters sit distinguishes the methods: low-rank matrices beside existing weights, small bottleneck layers between blocks, or trainable vectors prepended to the key and value sequences.
3. Because the optimiser state and gradients cover only the new parameters, memory during training falls by orders of magnitude rather than by a fraction.
4. The trained delta is a small artefact, so one base model can serve many task-specific deltas instead of many full copies.
5. At inference some methods fold the delta back into the base weights and add no latency, while others keep extra modules in the forward pass and do.

Example: Where the memory actually goes

Full fine-tuning of a 7B-parameter model in 16-bit needs about 14GB for weights, another 14GB for gradients, and roughly 28GB for Adam's two optimiser states — around 56GB before activations. Freeze the base and train 0.1% of the parameters, 7M of them, and gradients plus optimiser state cover 7M instead of 7B: about 0.08GB. The 14GB of frozen weights still has to be resident, so the total lands near 14GB rather than 56GB. The saving is in the optimiser, not the model, which is why it is so large.

Commonly confused with

PEFT is the family; LoRA is one member of it, alongside adapters, prefix tuning and prompt tuning. Using the two words interchangeably obscures a real choice, since the methods differ in where they insert parameters and whether they can be merged back into the base weights. All of them sidestep the memory problem that makes full fine-tuning expensive.

When to use it

Reach for it when:

• Adapting a model to a task or domain on hardware that cannot hold full fine-tuning state.
• Many task-specific variants are needed from one base model.
• Pretrained general capability must be preserved rather than overwritten.

Reach for something else when:

• The target behaviour differs fundamentally from pretraining, where a small delta cannot carry the change.
• Prompting or retrieval would solve it, which is cheaper than any training.
• Serving latency is critical and the chosen method cannot be merged into the base weights.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Parameter-Efficient Fine-Tuning (PEFT) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path