BF16 (Brain Floating Point 16) — AI Glossary

Reference details

Related terms

• Mixed Precision Training
• Quantization
• NVIDIA H100

How it works

1. The format spends its 16 bits as 1 sign, 8 exponent and 7 mantissa bits, which is the same exponent width as single precision and fewer mantissa bits than the alternative half-precision format.
2. Matching the exponent range means the representable magnitude span is the same as 32-bit float, so values that are representable in 32 bits do not overflow or underflow when cast down.
3. That is what removes loss scaling. The older half-precision format has a narrow exponent range in which small gradients round to zero, so training required multiplying the loss by a constant to shift gradients into range and dividing it out afterwards.
4. The cost is precision per value: fewer mantissa bits mean coarser spacing between representable numbers, so individual arithmetic results are less exact.
5. Training tolerates that because gradient estimates are already noisy from mini-batching, and the accumulation of many small updates averages the representation error out — while an overflow or an underflow does not average out.
6. Accumulation still happens in 32 bits inside the hardware's matrix units, so the format applies to the values held in memory rather than to the arithmetic performed on them.

Example: Trading precision for range, quantified

Counting the implicit leading bit, this format carries 7 + 1 = 8 bits of significand, so the relative spacing between adjacent representable values is 2^-8 = 0.0039, about 0.39%. The alternative half-precision format carries 10 + 1 = 11 bits, giving 2^-11 = 0.000488, about 0.049% — eight times finer, since 0.0039 / 0.000488 = 8. That is the loss. The gain is on the other axis: half precision's smallest normal magnitude is around 6.1 x 10^-5, so a gradient of 1 x 10^-6 underflows to zero and the parameter it belonged to receives no update at all, while the same value is comfortably representable here because the exponent range matches 32-bit float. The trade is therefore 8x coarser values against the elimination of silent zeroing, and it resolves in favour of range because a 0.39% representation error on a gradient is far below the noise that mini-batch sampling already introduces, whereas a gradient quietly becoming zero is not an error that averages away — it is a parameter that stops learning.

Commonly confused with

This is a storage and transport format, not a quantisation scheme, and conflating them misdirects effort. Quantisation converts a trained model's weights to a low-precision integer representation for serving, accepting a quality cost to reduce memory and bandwidth. This format is a floating-point type used during training for the values held in memory, with 32-bit accumulation inside the arithmetic units and 32-bit master weights for the optimiser. A model trained in it is a full-precision model that happened to be computed efficiently; it has not been quantised, and quantising it afterwards is a separate decision with separate costs.

When to use it

Reach for it when:

• Training on any accelerator that supports it natively, where it is the default and the alternative needs loss scaling.
• Fine-tuning, where the range safety removes a class of failure that presents as mysteriously worse convergence.
• Inference where memory bandwidth is the constraint and integer quantisation's quality cost is not acceptable.
• Mixed-precision setups generally, paired with 32-bit optimiser state and master weights.

Reach for something else when:

• On hardware without native support, where it is emulated and the throughput advantage disappears.
• Numerically sensitive reductions — normalisation statistics, loss accumulation — which belong in 32 bits regardless of the storage format.
• Where the coarser significand genuinely matters, such as some scientific workloads with tight error budgets.
• As a substitute for quantisation when serving memory is the constraint, since 16 bits per weight is still 16 bits.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to BF16 (Brain Floating Point 16) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path