Adapter (PEFT) — AI Glossary

Reference details

Related terms

• LoRA (Low-Rank Adaptation)
• Prefix Tuning
• Supervised Fine-Tuning (SFT)

How it works

1. A bottleneck module is inserted into each transformer block: a down-projection to a small dimension, a nonlinearity, and an up-projection back to the model width.
2. Only those modules are trained and the base weights stay frozen, so the gradient and optimiser footprint is proportional to the inserted parameters rather than the model.
3. The up-projection is initialised near zero and the module is wrapped in a residual connection, which makes the untrained adapter an approximate identity and training start from the base model's behaviour.
4. Because the module sits in series within the block, it adds computation to every forward pass — which is the structural difference from a low-rank update that can be folded into the existing weights.
5. That series placement is also what cannot be merged away: an adapter's latency cost is permanent while a merged low-rank update's is zero.
6. In exchange, several adapters can be held beside one resident base model and selected per request, so switching behaviour is loading a few megabytes rather than a second model.

Example: One base model, many behaviours

Keep one frozen base in memory and load a small adapter per task — support tone, code review, summarisation. Swapping tasks means swapping a few megabytes rather than loading a new multi-gigabyte model. That is the operational argument for adapters, independent of training cost.

Commonly confused with

Adapters and LoRA are related but not identical. Classic adapters insert new bottleneck layers into each block; LoRA adds low-rank updates to existing weight matrices and can be merged back into the base at inference. LoRA is the more common form today, and adapter is often used loosely to mean both.

When to use it

Reach for it when:

• Several task variants must share one base model in memory
• Adapters need independent versioning and rollback
• Full fine-tuning is not affordable

Reach for something else when:

• The change is broad and general rather than task-specific
• You have not tried prompting or retrieval first
• There is no evaluation to catch a quietly worse adapter

Primary source

• Houlsby et al., Parameter-Efficient Transfer Learning for NLP (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Adapter (PEFT) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path