ModelRefs / Batch Size — AI Glossary

Batch Size — AI Glossary

The number of independent inference requests processed simultaneously in a single GPU forward pass.

Overview

Larger batch sizes improve GPU utilization and throughput (tokens/s) but increase memory pressure and per-request latency. Continuous batching dynamically adds new requests to in-flight batches at the token level, improving GPU utilization for heterogeneous request lengths. Optimal batch size depends on model size, available VRAM, and latency SLO.

Reference details

Topic	inference
Last reviewed	2026-06-24

How it works

A batch is the set of sequences processed together in one forward pass, and the reason batching helps is that the weights are read once for the whole batch rather than once per sequence.
During decode the work is memory-bandwidth-bound, so that single weight read is the dominant cost and adding sequences to the batch is close to free until arithmetic becomes the constraint.
Throughput therefore rises steeply with batch size at first and then flattens, and the flattening point is where the workload stops being bandwidth-bound.
Per-request latency moves the other way, because a larger batch means each step does more work and, under window-based batching, arriving requests wait for the window to close.
Continuous batching changes the shape of the trade by admitting and retiring sequences between decoding steps, so a finished short sequence frees its slot instead of holding it until the longest member completes.
What caps the batch in practice is not compute but the key-value cache, which is allocated per sequence and scales with context length — so the achievable batch size is a function of how long the contexts are.

Example: Why the cache, not the compute, sets the batch ceiling

Take a 7B model with 32 layers and width 4,096, served at 4-bit so the weights are 7e9 x 0.5 = 3.5e9 bytes, about 3.5GB. The key-value cache holds a key and a value per layer per token: at 16 bits that is 2 x 2 x 32 x 4,096 = 524,288 bytes per token, and at an 8,192-token context 524,288 x 8,192 = 4,294,967,296 bytes, about 4.3GB for one sequence. On an 80GB device the weights leave 80 - 3.5 = 76.5GB, so the batch ceiling is 76.5 / 4.3 = 17 concurrent sequences at full context — not a compute limit, and not a number anyone would guess from the model's size. Halve the context to 4,096 and the per-sequence cache halves to about 2.1GB, so the ceiling roughly doubles to 76.5 / 2.1 = 36. That is why context length and concurrency are one decision rather than two, and why cache-reduction techniques — grouped-key attention, cache quantisation, paged allocation that avoids reserving the full context up front — buy throughput that no amount of additional compute would.

Commonly confused with

Serving batch size and training batch size are different quantities with the same name. In training, batch size is a hyperparameter affecting the optimisation itself — gradient noise, learning-rate scaling, final model quality — and is chosen for statistical reasons as well as hardware ones. In serving it affects only throughput and latency; the model's outputs are unchanged. Reading advice about one as advice about the other is a common source of confusion, particularly since training guidance frequently argues for larger batches on grounds that have no serving analogue.

When to use it

Reach for it when:

Throughput-oriented serving, where larger batches amortise the weight read across more sequences.
With continuous batching, which captures most of the available benefit without window-based latency penalties.
Sized against the cache budget at your actual context length, which is what sets the real ceiling.
Tuned by measurement on your own traffic, since the flattening point depends on model, hardware and request mix.

Reach for something else when:

Maximised blindly in latency-sensitive serving, where per-request time degrades past the throughput plateau.
Sized from compute capacity, which is almost never the binding constraint during decode.
With a window-based batcher at low arrival rates, where requests wait for a batch that never fills.
Reasoned about using training guidance, which concerns optimisation properties serving does not have.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Batch Size — AI Glossary.

Frequently asked questions

What is Batch Size?