GPU Utilization (MFU) — AI Glossary

Reference details

Related terms

• NVIDIA H100
• Throughput
• Distributed Inference

How it works

1. MFU is defined as achieved FLOPs divided by the device's peak FLOPs at the same numeric precision. Both terms must share a precision, or the ratio is meaningless.
2. Achieved FLOPs are computed analytically from the model and the workload, not measured by a profiler: for a transformer, roughly 6 x parameters x tokens for a training step, which counts forward, backward and the weight gradient.
3. The denominator is the vendor's peak for that precision, which assumes perfectly fed tensor cores. Nothing reaches it, so MFU is a fraction by construction and 100% is not the target.
4. The gap is accounted for by specific losses: collective communication between devices, pipeline bubbles while stages wait, data loading, kernel launch overhead, and attention operations that are memory-bound rather than compute-bound.
5. Inference splits in two. Prefill processes the whole prompt at once and is compute-bound, so it can reach training-like MFU. Decode generates one token at a time and must read the entire weight matrix per token, so it is memory-bandwidth-bound and MFU is low no matter what you do.
6. This is why batching is the main inference lever: more sequences per weight read amortises the bandwidth cost, moving decode toward the compute-bound regime where MFU can rise.

Example: Why decode MFU is low for a reason you cannot optimise away

Take a device with 312 TFLOPS of BF16 compute and 2 TB/s of memory bandwidth — a ratio of 312 / 2 = 156 FLOPs available per byte read. Now take a 13B model in BF16: weights are 13e9 x 2 = 26GB. Generating one token for one sequence requires reading all 26GB once and performing roughly 2 x 13e9 = 26 GFLOPs. That is 26e9 / 26e9 = 1 FLOP per byte, against a device that offers 156. The work is bandwidth-starved by a factor of 156, so time per token is set by 26GB / 2TB/s = 13ms of reading, and MFU is approximately 26e9 / (312e12 x 0.013) = 0.64%. Batch 64 sequences and the same single weight read serves 64 tokens: 64 x 26 = 1,664 GFLOPs against the same 13ms, giving about 41% MFU. The weights were read once either way — that is the entire argument for continuous batching, and it explains why a single-user deployment of a large model is almost always wasting the GPU it is paying for.

Commonly confused with

MFU is not the utilisation percentage reported by GPU monitoring tools, and the gap routinely misleads capacity planning. The tool's figure says what fraction of sampled intervals had any kernel resident — it reads near 100% on a workload that is entirely memory-bound and doing almost no arithmetic. MFU says what fraction of the device's arithmetic capability is being used. A decode server can show 99% device utilisation and 2% MFU simultaneously, both correct. If you are deciding whether to buy more GPUs or batch better, MFU is the number that answers it.

When to use it

Reach for it when:

• Evaluating training efficiency, where MFU is the standard comparable figure across cluster sizes and parallelism strategies.
• Deciding whether a performance problem is compute, bandwidth or communication, since low MFU with high device utilisation localises it to bandwidth.
• Justifying batching, quantisation or attention-kernel work, each of which moves MFU in a predictable direction.
• Cost modelling, since FLOPs per currency unit is only meaningful against achieved rather than peak throughput.

Reach for something else when:

• As a target to maximise in interactive inference, where latency is the product requirement and large batches trade it away.
• Compared across precisions or hardware generations without stating the peak used as the denominator.
• As the figure to optimise during decode, which is bandwidth-bound by arithmetic you cannot change.
• In place of end-to-end throughput and latency. High MFU on an inefficient algorithm is efficient execution of unnecessary work.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to GPU Utilization (MFU) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path