Memory Bandwidth — AI Glossary

Reference details

Related terms

• GPU Memory (VRAM)
• Tokens Per Second (TPS)
• Quantization

How it works

1. Generating one token requires every weight in the model to be read from high-bandwidth memory into the compute cores, because each layer must be applied before the next can start.
2. At batch size one that read is amortised across a single sequence, so the arithmetic performed per byte fetched is tiny and the cores idle waiting on memory.
3. The resulting floor on per-token latency is the model's byte footprint divided by the device's bandwidth, and no amount of additional compute moves it.
4. Batching attacks the ratio rather than the bandwidth: one weight read serves every sequence in the batch, so aggregate throughput rises while per-step latency stays roughly constant.
5. Quantisation attacks the numerator, since halving the bytes per weight halves the bytes that must cross the bus for each token.
6. Speculative decoding attacks the count of reads, verifying several draft tokens in one pass over the weights so that a single traversal yields more than one accepted token.

Example: The decode ceiling is arithmetic, not tuning

Take a 70B model at fp16, so about 140 GB of weights, on an H100 SXM rated at 3.35 TB/s. One token needs one full pass over the weights: 140 / 3350 ≈ 0.0418 seconds, which is about 41.8 ms, giving roughly 23.9 tokens per second at batch size one. That number is a ceiling. A faster kernel, a better scheduler and a larger power budget do not improve it, because the device is not computing — it is reading. Three levers move it, and each moves a different term. Quantising to 8-bit halves the footprint to 70 GB: 70 / 3350 ≈ 0.0209 seconds, about 20.9 ms, so roughly 47.9 tokens per second, double the fp16 figure because the numerator halved. Batching leaves per-step latency at 41.8 ms but serves the whole batch from one weight read, so 32 concurrent sequences yield about 766 tokens per second in aggregate, which is 32 times the single-stream rate, while each individual user still waits 41.8 ms per token. And the device itself matters in direct proportion: the same fp16 model on an A100 at 2.0 TB/s gives 140 / 2000 = 0.07 seconds, exactly 70 ms, or about 14.3 tokens per second. The H100 advantage is 3.35 / 2.0 = 1.675, and 23.9 / 14.3 ≈ 1.67 reproduces it, because the ratio of speeds is just the ratio of bandwidths.

Commonly confused with

Memory bandwidth is not memory capacity, and the two constrain different failure modes. Capacity decides whether the model loads at all; bandwidth decides how fast it generates once loaded. A device with ample capacity and modest bandwidth will host a large model and decode slowly, which is the common disappointment when people size hardware by VRAM alone. The second distinction is between the two phases of inference: prefill processes the whole prompt in parallel and is compute-bound, so it scales with FLOPS, while decode is sequential and bandwidth-bound. This is why prompt processing and token generation respond to entirely different optimisations, and why a benchmark that reports only one of them describes half the system.

When to use it

Reach for it when:

• When sizing hardware for generation-heavy workloads, where bandwidth predicts tokens per second more directly than FLOPS does.
• To sanity-check a vendor throughput claim, since the footprint-over-bandwidth figure is a ceiling no software can exceed.
• When deciding whether quantisation will help, as the gain is proportional to the reduction in bytes per weight.
• In capacity planning for concurrency, because batching raises aggregate throughput without improving any single user's per-token latency.

Reach for something else when:

• As the constraint on prefill, which is compute-bound and scales with arithmetic rather than with bytes moved.
• For training, where gradients, activations and optimiser state change the balance and interconnect often binds first.
• To predict latency for small models that fit in cache, where the clean footprint-over-bandwidth model stops describing the hardware.
• As the only hardware criterion, since capacity still decides whether the weights and the KV cache fit at the intended context length.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Memory Bandwidth — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path