ModelRefs / Quantization — AI Glossary

Quantization — AI Glossary

Compressing model weights and activations to lower-precision formats (INT8, INT4, FP4) to reduce memory and accelerate inference.

Overview

GPTQ, AWQ, and GGUF are common post-training quantization schemes. INT4 typically loses 1–3 points on benchmarks while cutting memory 4× and improving throughput 2–3×. QLoRA combines quantization with fine-tuning for memory-efficient adaptation.

Reference details

Topic	inference
Last reviewed	2026-06-24

How it works

Weights are divided into groups — a row, or a block of some fixed width — and each group is examined independently rather than the tensor as a whole.
For each group a scale is computed from its range, and the floats in it are mapped onto a small set of integers, so what is stored is the integers plus that one scale.
Reconstruction multiplies the integer by the scale, which means the error on any weight is bounded by the group's step size and a group containing one extreme value quantises everything else badly.
Those outlier channels are the central problem, and schemes address them by keeping a small fraction of weights at higher precision, or by rotating the weight space so no single channel dominates.
Calibration data is run through the model to decide which weights matter most, which is why two quantisations of the same model at the same bit width can differ in quality.
At inference the integers are either expanded back to floating point just before each matrix multiply or consumed by an integer kernel directly, and the memory saving is real in both cases while the speed-up only arrives in the second.

Example: Where the memory goes

A 7B-parameter model at FP16 needs roughly 7e9 × 2 bytes ≈ 14 GB for weights alone, before the KV cache. At INT4 that is about 3.5 GB, which is the difference between needing a data-centre GPU and fitting on a consumer card. Quality loss is usually small but is never zero, and it lands unevenly across tasks.

weights_bytes = params * bytes_per_param
FP16: 7e9 * 2   = 14.0 GB
INT8: 7e9 * 1   =  7.0 GB
INT4: 7e9 * 0.5 =  3.5 GB   # KV cache is on top of this

Commonly confused with

Quantization is not distillation or pruning. The architecture and parameter count are unchanged — only the numeric precision of the stored values differs. Distillation trains a smaller model; pruning removes weights; quantization keeps every weight at lower precision.

When to use it

Reach for it when:

Memory is the binding constraint on which model you can serve
You are serving open-weight models on your own hardware
A small, measured quality loss is an acceptable trade for capacity

Reach for something else when:

You have not measured the loss on your own task — averages hide task-specific damage
The workload is precision-sensitive, such as exact numeric extraction
You are using a hosted API, where the provider already made this decision

Referenced by

This term is used by the following ModelRefs references:

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Quantization — AI Glossary.

Frequently asked questions

What is Quantization?