ModelRefs / Continuous Batching — AI Glossary

Continuous Batching — AI Glossary

An inference scheduling technique that swaps finished sequences out of the batch and new requests in at every decoding step.

Overview

Continuous batching keeps GPUs fully utilized under bursty request patterns — the key reason vLLM and SGLang achieve high throughput. Contrasts with static batching, which waits for all requests in a batch to finish before accepting new ones.

Reference details

Topic	inference
Last reviewed	2026-06-24

How it works

The scheduler operates per iteration rather than per batch, deciding at every decode step which sequences occupy the batch.
A sequence that emits its final token releases its slot immediately, rather than waiting for the longest sequence in its cohort.
A queued request takes that slot on the next iteration, so arrival does not have to align with a batch boundary.
Utilisation therefore tracks demand instead of the slowest member, which is where the throughput gain comes from.
It depends on per-sequence KV cache that can be allocated and released independently, which is why it pairs with paged cache management.

Example: The cost of waiting for the longest sequence

Eight requests batched statically, seven generating 100 tokens and one generating 1,000. The batch occupies its slots for 1,000 steps. Seven slots finish at step 100 and sit idle for 900 — 6,300 idle slot-steps out of 8,000, so about 79% of the capacity is wasted. Continuous batching frees each slot at step 100 and admits new work, so those 6,300 slot-steps do something. The gain is entirely in the variance of output length, which is why it matters most for chat and least for uniform batch jobs.

Commonly confused with

Static batching waits for every sequence in a batch to finish, so one long generation holds short ones hostage and the slot sits idle. Continuous batching evicts a sequence the moment it completes and admits a waiting request in its place, at every decoding step. The gain is largest exactly where static batching is worst: mixed workloads with widely varying output lengths.

When to use it

Reach for it when:

Serving concurrent requests whose output lengths vary, which is the normal case for chat.
Requests arrive continuously rather than in aligned batches.
Throughput per device is the constraint being optimised.

Reach for something else when:

Output lengths are uniform, where static batching loses little.
A single request at a time, where there is no batch to schedule.
Explaining single-request latency, which this does not improve.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Continuous Batching — AI Glossary.

Frequently asked questions

What is Continuous Batching?