Throughput — AI Glossary

Reference details

Related terms

• Latency
• Inference
• Continuous Batching
• vLLM

How it works

1. Concurrent requests are grouped so that a single read of the model weights from memory serves every sequence in the group, which is the mechanism the whole metric rests on.
2. Because decode is bound by memory bandwidth rather than arithmetic, the time for one step is roughly fixed regardless of how many sequences are in it.
3. Aggregate throughput is therefore batch size divided by step time, and it rises almost linearly with batch size while that fixed step time holds.
4. Continuous batching keeps the group full by admitting and evicting sequences at every iteration rather than per batch, so a finished sequence frees its slot immediately.
5. The ceiling is set by KV cache memory: each concurrent sequence holds its own cache, and the batch can only grow until that memory is exhausted.
6. Past that point the scheduler must preempt, and preemption evicts a sequence's cache so its prefill is recomputed when it resumes — so a saturated server performs strictly more work per token delivered than the same server below saturation.

Example: The trade you are actually making

Batch size 1 might give 30 tokens/second for one user with minimal wait. Batch 32 might give 600 tokens/second aggregate while each user waits longer to be scheduled. Total work rose; individual experience got worse. That is why throughput and latency are reported together — optimising one in isolation always degrades the other.

Commonly confused with

Throughput is not latency and improving one usually costs the other. Throughput is system-wide work per unit time; latency is what a single user waits. A serving setup can be excellent at one and unacceptable at the other.

When to use it

Reach for it when:

• Sizing capacity and cost per token for a workload
• Offline or batch jobs where nobody is waiting on an individual response
• Comparing serving configurations under a fixed latency budget

Reach for something else when:

• As the headline metric for interactive products — users feel latency, not throughput
• Quoted without batch size and context length, where the number means nothing
• As a proxy for cost; utilisation and hardware pricing decide that

Referenced by

This term is used by the following ModelRefs references:

• streaming inference stack
• batch inference stack

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Throughput — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path