vLLM — AI Glossary

Reference details

Related terms

• PagedAttention
• Continuous Batching
• Inference

How it works

1. The KV cache is divided into fixed-size blocks instead of one contiguous reservation per sequence, which is the idea PagedAttention takes from operating-system paging.
2. Each sequence holds a block table mapping its logical token positions to physical blocks, so the blocks it occupies need not be adjacent in memory.
3. Because allocation is per block rather than per worst-case sequence length, memory is not reserved for tokens a request may never generate, and fragmentation largely disappears.
4. The attention kernel is written to gather keys and values through that block table, so the indirection costs a lookup rather than a copy.
5. Sequences sharing a prefix — a common system prompt, or several samples from one prompt — point at the same physical blocks, and a block is copied only when one of them writes to it.
6. The scheduler then runs per iteration rather than per batch, admitting newly arrived requests and releasing the blocks of finished ones at every step, which is what keeps the accelerator from idling on completed slots.

Example: Why continuous batching matters, measured as slot utilisation

With static batching, a batch of eight requests waits for the slowest to finish before any slot frees. Continuous batching evicts each sequence as it completes and admits a new one immediately, so the GPU is not idling on finished slots. That scheduling difference, not a faster kernel, is where most of the throughput gain comes from. The waste is countable. Take eight requests generating 50, 100, 150, 200, 250, 300, 350 and 400 tokens. Under static batching every slot is held until the longest finishes, so the batch occupies 8 x 400 = 3,200 slot-steps while performing only 50 + 100 + … + 400 = 1,800 tokens of work — a utilisation of 1,800 / 3,200 ≈ 56.3%. Nearly half the capacity is spent holding finished slots open. Continuous batching releases each one at completion and refills it, which is why the gain is largest exactly where response lengths vary most.

Commonly confused with

vLLM is an inference server, not a model and not a framework for building applications. It serves open-weight models over an OpenAI-compatible API. It does not fine-tune, does not orchestrate, and is not an alternative to a gateway — those sit at different layers. It is also not a single-user runtime: its advantages are scheduling and memory advantages that only appear under concurrency, so benchmarking it with one request at a time measures almost none of what it was built for.

When to use it

Reach for it when:

• Self-hosting open-weight models with real concurrency
• You need PagedAttention or LoRA hot-swap at serving time
• You want an OpenAI-compatible surface over your own hardware
• Where response lengths vary widely, which is exactly where the 56.3% utilisation above is recovered

Reach for something else when:

• You use hosted APIs — the provider already runs something like it
• Single-user or batch workloads, where a simpler runtime is easier to operate
• Nobody owns GPU operations; a serving stack is infrastructure to run
• Benchmarked one request at a time, which measures none of its scheduling advantage

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to vLLM — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path