GPTQ — AI Glossary

Reference details

Related terms

• AWQ (Activation-Aware Weight Quantization)
• Quantization
• GGUF

How it works

1. Quantisation proceeds one layer at a time, with the objective of reproducing that layer's output rather than minimising weight error in isolation.
2. Second-order information about the layer's inputs is used to decide which weight errors matter, since an error on a weight that meets large inputs costs more.
3. Weights are quantised in sequence, and after each one the remaining unquantised weights are adjusted to compensate for the error just introduced.
4. That compensation is the mechanism: error is not merely accepted per weight, it is partly absorbed by the weights that follow.
5. A calibration set supplies the input statistics, and like other post-training methods it needs no gradients or training run.

Example: Compensation is what separates it from naive rounding

Round a weight from 0.37 to the nearest 4-bit grid point, say 0.375, and the residual is -0.005. Naive quantisation discards that; this approach pushes it into the not-yet-quantised weights in the same row, so the layer's output error is smaller than the sum of individual weight errors would suggest. Over a layer with millions of weights the residuals partly cancel rather than accumulate — which is why sequential-with-compensation beats rounding everything independently at the same bit width.

Commonly confused with

GPTQ and AWQ both produce near-lossless 4-bit weights after training, and differ in how they decide what to protect. GPTQ reconstructs layer outputs using second-order information about the weights; AWQ identifies the small fraction of weights the activations actually depend on and keeps those at higher precision. Neither is a format — GGUF is a container, and either method's output can end up in one.

When to use it

Reach for it when:

• 4-bit weights are needed with minimal output degradation and no training budget.
• Calibration data representative of the deployment distribution is available.
• The serving stack has kernels for the format.

Reach for something else when:

• Numerical precision is critical to the application.
• No calibration data exists, since the second-order statistics come from it.
• Choosing between this and the activation-aware alternative without measuring both on your own model — the better choice is empirical.

Primary source

• Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to GPTQ — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path