AWQ (Activation-Aware Weight Quantization) — AI Glossary

Reference details

Related terms

• GPTQ
• Quantization
• GGUF

How it works

1. A small calibration set is run through the model to collect activation statistics per channel, so the method observes what the weights actually meet at inference.
2. Channels whose activations are large are identified as the ones where weight error does most damage to the output.
3. Those channels' weights are scaled up before quantisation and the inverse scale is folded elsewhere, so they land in a part of the integer grid where relative error is smaller.
4. Quantisation to 4-bit then proceeds per group with its own scale, which keeps the dynamic range local rather than global.
5. No gradients are involved — it is a post-training transform, which is why it runs in minutes rather than requiring a training budget.

Example: Where 4-bit memory saving comes from

A 13B-parameter model in 16-bit holds about 13e9 x 2 = 26GB of weights. At 4 bits that is 13e9 x 0.5 = 6.5GB, a quarter, plus a small overhead for per-group scales — at a group size of 128 and one 16-bit scale per group, about 13e9/128 x 2 ≈ 0.2GB. So roughly 6.7GB against 26GB, which is the difference between needing a 40GB accelerator and fitting comfortably on a 24GB one. The quality question is separate; the memory arithmetic is what motivates it.

Commonly confused with

The counterpart to GPTQ. Where GPTQ works from curvature information about the weights themselves, this uses activation statistics to find which weights matter for real inputs, which is why it needs a calibration set drawn from representative data. Both are post-training methods: neither requires retraining, and both are distinct from quantization-aware training, which changes the training run.

When to use it

Reach for it when:

• A model must fit in less memory with minimal quality loss and no training budget.
• Inference is memory-bandwidth bound, where fewer bytes per weight directly raises token rate.
• Activation statistics from representative data are available for calibration.

Reach for something else when:

• Full precision is required for numerical sensitivity.
• No representative calibration data exists, since the method depends on observed activations.
• The serving stack lacks optimised kernels for the format, where the memory saving does not convert to speed.

Primary source

• Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to AWQ (Activation-Aware Weight Quantization) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path