Distributed Inference — AI Glossary

Reference details

Related terms

• Pipeline Parallelism
• Model Parallelism
• GPU Memory (VRAM)

How it works

1. A model that does not fit in one device's memory is partitioned. Tensor parallelism splits individual weight matrices across devices; pipeline parallelism assigns whole layers to different devices.
2. Under tensor parallelism every device holds a slice of the same layer and computes a partial result, so each layer ends in a collective reduction to combine them.
3. That collective runs for every layer of every token, which is why interconnect bandwidth rather than arithmetic usually sets the ceiling.
4. Under pipeline parallelism a token's activations traverse devices in sequence, so devices idle unless several requests are in flight to fill the stages.
5. The KV cache is partitioned along with the weights, so cache capacity — and therefore how many concurrent requests fit — scales with the group rather than with one device.

Example: Where the parallel efficiency goes

A model needs 160GB of weights and the available accelerators hold 80GB each, so a minimum of two devices, before any KV cache. Split across two with tensor parallelism, each layer's compute halves but each layer gains a reduction over the interconnect. If that collective costs 15% of the layer's time, two devices deliver about 2 × 0.85 = 1.7× the throughput, not 2×. Add two more and the collective spans four devices and costs more, so the gap from linear widens — which is why fast interconnect is a capability rather than a nicety.

Commonly confused with

Conflated with batching and with data parallelism, which solve different problems. Data parallelism puts a full copy of the model on each device and sends different requests to each: more throughput, no help with a model that does not fit. Distributed inference splits one model across devices so a single forward pass spans them, which is what makes an oversized model runnable at all and costs collective communication per layer. Batching is orthogonal to both — it raises utilisation within whatever topology you have. The ordering in practice: fit the model first, then batch, then replicate.

When to use it

Reach for it when:

• Model weights plus KV cache exceed a single device's memory.
• You need more KV-cache headroom for concurrency than one device can hold.
• Latency targets require more memory bandwidth than one device provides.

Reach for something else when:

• The model already fits — replication and batching will beat partitioning on both throughput and complexity.
• The interconnect is slow, where per-layer collectives can erase the gain entirely.
• Quantisation would make it fit on one device at acceptable quality, which is the cheaper fix to try first.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Distributed Inference — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path