QLoRA — AI Glossary

Reference details

Related terms

• LoRA (Low-Rank Adaptation)
• Quantization
• Parameter-Efficient Fine-Tuning (PEFT)

How it works

1. The base model is loaded in 4-bit precision using a normal-float format designed for the roughly normal distribution of trained weights.
2. Those base weights stay frozen and quantised, so they consume a quarter of the memory of 16-bit and receive no gradients.
3. Low-rank adapters are attached in higher precision, and only those are trained — so gradients and optimiser state cover the adapters alone.
4. Quantisation constants are themselves quantised, recovering a further slice of memory from the per-block scales.
5. Optimiser state can be paged to host memory when a spike would otherwise overflow, which is what turns an out-of-memory crash into a slowdown.

Example: The arithmetic that put 65B on one card

A 65B model in 16-bit is about 65e9 x 2 = 130GB of weights alone, before gradients or optimiser state — several accelerators. At 4 bits the weights are 65e9 x 0.5 ≈ 32.5GB. Adapters at 0.1% of parameters add 65e6 x 2 ≈ 0.13GB, and their optimiser state a similar order. The total lands near 33GB: one 48GB card, where full fine-tuning needed well over 500GB. The quality claim is separate and empirical; the memory claim is division.

Commonly confused with

LoRA applied on top of a base model held in 4-bit precision, which is what makes single-GPU fine-tuning of large models feasible. The consequence people miss is that quantization does not disappear afterwards: the adapters were trained against quantized weights, so merging them into a full-precision base is not guaranteed to behave like the model you trained. Evaluate the deployment configuration, not the training one.

When to use it

Reach for it when:

• Fine-tuning a large model on a single accelerator that could not hold it in higher precision.
• Memory is the binding constraint and some inference-time overhead is acceptable.
• Many task-specific adapters are wanted over one quantised base.

Reach for something else when:

• Full precision is needed for numerical sensitivity.
• Serving latency is critical — dequantisation on the forward pass costs throughput.
• Memory is already sufficient, where ordinary low-rank adaptation avoids the quantisation entirely.

Primary source

• Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to QLoRA — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path