Pipeline Parallelism — AI Glossary

Reference details

Related terms

• Model Parallelism
• Distributed Inference
• Disaggregated Serving

How it works

1. Consecutive layers are assigned to different devices, so each holds a contiguous slice of the network — stage one the first layers, stage two the next, and so on.
2. A request's activations pass point-to-point from stage to stage, which is a single transfer per boundary rather than a collective inside every layer.
3. With one request in flight, every stage but one is idle while it waits. That idleness is the pipeline bubble, and it is the method's defining cost.
4. The fix is to keep several micro-batches moving at once, so stage one works on batch three while stage four finishes batch one.
5. Per-device memory falls roughly as 1/N because each device stores only its own layers, which is what makes an otherwise unservable model fit.

Example: The bubble, and why micro-batching is not optional

Four stages, each taking 10ms. One request end to end is 40ms, during which each stage works 10ms and idles 30ms — 25% utilisation. Push eight micro-batches through and the pipeline fills: total time is roughly (8 + 4 - 1) x 10 = 110ms for eight batches, or 13.75ms each, against 40ms run one at a time. Utilisation rises to about 8/11, near 73%. The arithmetic is why pipeline parallelism is a throughput technique and never a latency one.

Commonly confused with

Pipeline parallelism splits by layer, tensor parallelism splits inside each layer. That determines the network bill: pipeline passes activations point-to-point between stages and tolerates slower links, while tensor parallelism synchronises within every layer and effectively demands a fast intra-node interconnect. Pipeline's cost is idle stages waiting for work, which micro-batching reduces but does not remove.

When to use it

Reach for it when:

• A model must be split across devices and the interconnect between them is slow, since point-to-point transfers tolerate what collectives will not.
• Devices sit in separate nodes, where tensor parallelism's per-layer synchronisation would dominate.
• Throughput matters more than single-request latency, so the bubble can be filled.

Reach for something else when:

• Single-request latency is the target — the bubble makes one request slower, not faster.
• Fast intra-node interconnect is available, where tensor parallelism usually wins.
• Traffic is too thin to keep several micro-batches in flight, leaving the bubble unfilled.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Pipeline Parallelism — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path