Disaggregated Serving — AI Glossary

Reference details

Related terms

• Chunked Prefill
• KV Cache
• Pipeline Parallelism

How it works

1. Inference splits into two phases with opposing resource profiles. Prefill processes the whole prompt in parallel and is compute-bound; decode emits one token at a time, re-reading the weights for each, and is memory-bandwidth-bound.
2. Co-locating them forces one hardware choice for both and creates interference: a long prefill occupies the device and stalls the decode steps of other requests, which is what produces the latency spikes users notice.
3. Disaggregation assigns the phases to separate pools — compute-dense accelerators for prefill, bandwidth-optimised ones for decode — so each is provisioned against the resource it actually consumes.
4. The KV cache produced by prefill must then move to the decode pool over the interconnect. That transfer is the cost of the architecture and the reason it needs a fast fabric to be worth doing.
5. Each pool scales on its own signal: prefill capacity follows prompt length and arrival rate, decode capacity follows concurrent sequences and their output lengths.
6. The separation also decouples the latency metrics. Time-to-first-token is a prefill-pool property and inter-token latency is a decode-pool property, so each can be tuned without trading against the other.

Example: Why the phases want different hardware, in one ratio

Take a device with 312 TFLOPS of 16-bit compute and 2 TB/s of memory bandwidth: it offers 312e12 / 2e12 = 156 FLOPs for every byte read. Prefill on a 2,000-token prompt for a 13B model performs about 2 x 13e9 x 2,000 = 5.2e13 FLOPs while reading the 26GB of weights once, giving 5.2e13 / 26e9 = 2,000 FLOPs per byte — well above the device's 156, so prefill is compute-bound and the device's arithmetic is the constraint. Decode for a single sequence performs about 2 x 13e9 = 2.6e10 FLOPs per token against the same 26GB read: 2.6e10 / 26e9 = 1 FLOP per byte, a factor of 156 below what the device offers, so decode is bandwidth-starved and most of the arithmetic sits idle. The two phases differ in arithmetic intensity by 2,000 / 1 = 2,000x on identical hardware. That is the entire case for separating them — no single device configuration is right for both, and co-location means one phase always wastes the resource the other needs.

Commonly confused with

Disaggregated serving is not continuous batching, though both target utilisation. Continuous batching improves decode efficiency by letting many sequences share each weight read, raising arithmetic intensity within one pool. Disaggregation separates the phases onto different hardware so each can be provisioned correctly. They compose — a decode pool still batches continuously — and continuous batching is the thing to implement first, since it is a configuration change rather than an architecture with a KV-transfer path to build and operate.

When to use it

Reach for it when:

• Large-scale serving where prefill and decode pools are each big enough to be provisioned and scaled independently.
• Mixed traffic combining long prompts with long generations, which is where co-located interference is worst.
• Strict and separate latency targets for time-to-first-token and inter-token latency.
• Clusters with a fast interconnect, without which the KV transfer erodes the gain.

Reach for something else when:

• Small deployments, where splitting a handful of accelerators into two pools leaves both too small to utilise.
• Before continuous batching is in place, which captures most of the available decode efficiency for far less operational complexity.
• Without fast inter-pool bandwidth, since moving the KV cache per request is the architecture's recurring cost.
• Uniform short-prompt, short-output traffic, where the phases do not diverge enough to repay the complexity.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Disaggregated Serving — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path