ModelRefs / Mixed Precision Training — AI Glossary

Mixed Precision Training — AI Glossary

Training using low-precision (FP16/BF16) for forward/backward passes and full-precision (FP32) for optimizer states, balancing speed and stability.

Overview

Mixed precision (NVIDIA AMP, PyTorch autocast) halves memory and doubles throughput versus pure FP32 by storing activations and weights in 16-bit. A FP32 master copy of weights is maintained for stable optimizer updates. Loss scaling (for FP16) or BF16 (no scaling needed) prevents gradient underflow. Universal practice for modern LLM training.

Reference details

Topic	training
Last reviewed	2026-06-24

How it works

Forward and backward passes run in a 16-bit format, so matrix multiplications execute on tensor cores at roughly double the FP32 rate and activations occupy half the memory.
A full-precision master copy of the weights is kept, because an optimiser step adds a tiny update to a large weight and 16-bit arithmetic rounds that update away. Updates are applied to the master copy, which is then cast down for the next forward pass.
Operations that are numerically fragile stay in 32-bit regardless: normalisation statistics, softmax accumulation, loss reduction, and anything summing many terms.
FP16 has a narrow exponent range, so small gradients underflow to zero. Loss scaling multiplies the loss by a large constant before the backward pass and divides it out of the gradients afterwards, moving them into representable range.
BF16 trades mantissa bits for exponent range, matching FP32's range. Underflow stops being the problem, loss scaling becomes unnecessary, and the cost is lower precision per value — which is why BF16 is the default on hardware that supports it.
The speedup comes from tensor-core throughput and reduced memory traffic. Any step that falls back to 32-bit, or any layout change forcing a cast, gives some of it back.

Example: What mixed precision does not halve

Count bytes per parameter for Adam. In pure FP32: 4 for the weight, 4 for the gradient, 4 each for the two optimiser moments — 4 + 4 + 4 + 4 = 16 bytes. In mixed precision with an FP32 master copy: 2 for the BF16 weight, 2 for the BF16 gradient, 4 for the master weight, and 4 each for the two FP32 moments — 2 + 2 + 4 + 4 + 4 = 16 bytes. Identical. For a 7B model that is 7e9 x 16 = 112GB of optimiser and weight state either way, and the common belief that mixed precision halves training memory is wrong about this part of it. What it does halve is activation memory, which is the term that scales with batch size and sequence length and is usually what actually bounds your batch: activations stored in 2 bytes instead of 4. So the correct expectation is roughly 2x throughput and a larger feasible batch, with the fixed per-parameter state unchanged — and if your model does not fit at all, mixed precision is not the lever. Sharding the optimiser state is.

Commonly confused with

Mixed precision is training, quantisation is inference, and they are frequently conflated because both involve fewer bits. Mixed precision keeps a full-precision master copy and uses low precision only for throughput during computation, so the trained result is a full-precision model. Quantisation converts a finished model's weights to low precision permanently, accepting a quality cost to reduce serving memory and bandwidth. Training in 16 bits does not produce a quantised model, and a quantised model has not been trained in mixed precision — the one place they meet is quantisation-aware training, which simulates the inference format during the forward pass.

When to use it

Reach for it when:

Essentially all modern training on hardware with tensor cores, where it is the default rather than an optimisation.
BF16 wherever the hardware supports it, since matching FP32's exponent range removes loss scaling and its failure modes.
When activation memory is the constraint on batch size, which is the term mixed precision genuinely halves.
Fine-tuning, where throughput translates directly into iteration speed on a short run.

Reach for something else when:

Expecting it to halve total training memory. Per-parameter weight and optimiser state is unchanged; only activations shrink.
As a fix for a model that does not fit. That needs optimiser sharding, offload or parameter-efficient methods.
FP16 without loss scaling, which silently zeroes small gradients and presents as a model that trains to a worse loss for no visible reason.
On numerically sensitive reductions forced into 16-bit. Normalisation and loss accumulation belong in 32-bit, and overriding that to chase throughput produces divergence.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Mixed Precision Training — AI Glossary.

Frequently asked questions

What is Mixed Precision Training?