ModelRefs / LoRA (Low-Rank Adaptation) — AI Glossary
LoRA (Low-Rank Adaptation) — AI Glossary
A PEFT method that injects small trainable rank-decomposition matrices into frozen model weight layers. LoRA is not quantization and the two are independent.
Overview
LoRA cuts trainable parameters by 100–1,000× and produces tiny (MB-scale) adapter files that can be hot-swapped at inference time or merged into the base model. QLoRA combines LoRA with 4-bit quantization to fine-tune 65B models on a single consumer GPU.
Reference details
| Topic | training |
|---|---|
| Also known as | Low-Rank Adaptation |
| Last reviewed | 2026-06-24 |
Related terms
How it works
- The chosen weight matrices are frozen and a pair of small factors is attached alongside each, so the forward pass computes the original product plus the product through the pair.
- The second factor is initialised to zero and the first at random, which makes the adapter an exact identity at step zero — training therefore starts from the base model's behaviour rather than from a perturbation of it.
- Only those factors receive gradients, so optimiser state is proportional to the adapter rather than to the model, and that is where most of the memory saving comes from rather than from the parameter count itself.
- The adapter's contribution is scaled by a constant over the rank, which decouples how much the adaptation is weighted from how many parameters it has — so changing rank does not implicitly change the learning rate.
- Which modules are adapted is a choice: attention projections only is cheap and usually sufficient, while adapting every linear layer costs more and helps on larger behavioural shifts.
- At serving the factors can be folded into the frozen weights for zero added latency, or kept separate so several adapters share one resident base model and are switched per request.
Example: Why the adapter is so small
A weight matrix of 4096 × 4096 holds about 16.8M parameters. LoRA freezes it and trains two matrices of 4096 × 8 and 8 × 4096 instead — roughly 65.5K parameters, about 0.4% of the original. That is the whole idea: the update is assumed to be low-rank, so you train the small factors and leave the large matrix alone.
# rank r = 8 on one 4096 x 4096 layer
full = 4096 * 4096 # 16,777,216 trainable
lora = 4096 * 8 + 8 * 4096 # 65,536 trainable (~0.39%)Commonly confused with
LoRA is not quantization and the two are independent. LoRA reduces how many parameters you train; quantization reduces the precision each weight is stored in. QLoRA is simply both at once — a 4-bit base model with LoRA adapters trained on top.
When to use it
Reach for it when:
- You want several task-specific variants sharing one base model
- Adapters need to be swapped, versioned or rolled back independently
- Full fine-tuning will not fit in available memory
Reach for something else when:
- The change is large and general — a big domain shift may need full fine-tuning
- Prompting or retrieval has not been tried; this is the more expensive lever
- You cannot evaluate the result; a quietly worse adapter is easy to ship
Referenced by
This term is used by the following ModelRefs references:
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LoRA (Low-Rank Adaptation) — AI Glossary.
Frequently asked questions
What is LoRA (Low-Rank Adaptation)?
A PEFT method that injects small trainable rank-decomposition matrices into frozen model weight layers.
Is LoRA (Low-Rank Adaptation) the same as Low-Rank Adaptation?
Yes — Low-Rank Adaptation are common aliases for LoRA (Low-Rank Adaptation).
What concepts are related to LoRA (Low-Rank Adaptation)?
Closely related concepts include peft, fine tuning, quantization.