Tensor Parallelism — AI Glossary

Reference details

Related terms

• Inference
• Throughput
• vLLM

How it works

1. Each weight matrix in a layer is sharded across devices, so every device holds a slice of every layer rather than whole layers.
2. A matrix multiply is split along a dimension chosen so each device can compute a partial result independently.
3. Those partials must be combined, so each sharded operation ends in a collective — an all-reduce or all-gather — across the whole group.
4. That collective runs twice per transformer block, for every token, which is why interconnect bandwidth rather than arithmetic sets the ceiling.
5. Because all devices work on the same token simultaneously, single-request latency improves — the property pipeline parallelism cannot offer.

Example: Why it needs a fast link and pipeline parallelism does not

A 40-layer model with two collectives per layer performs 80 collectives per token. At 50 microseconds each that is 80 x 0.05 = 4ms of pure communication per token, so a target of 20 tokens per second — 50ms per token — spends 8% of its budget on synchronisation. Move the same group onto a slower link where a collective costs 500 microseconds and it becomes 40ms per token, consuming 80% of the budget. Pipeline parallelism passes activations once per stage boundary instead, which is why it survives on links this one cannot use.

Commonly confused with

The counterpart to pipeline parallelism: it splits individual weight matrices rather than assigning whole layers, so every device works on every layer and they must synchronise inside each one. That gives lower latency than pipelining and much heavier communication, which is why it is normally kept within a single node. Both sit under model parallelism, which is the general term.

When to use it

Reach for it when:

• Single-request latency matters and the model must be split across devices.
• Devices share a fast intra-node interconnect.
• Per-device memory needs reducing across every layer rather than by assigning layers.

Reach for something else when:

• Devices are in separate nodes on a slower link, where per-layer collectives dominate.
• Throughput rather than latency is the goal, where pipeline parallelism or replication is cheaper.
• The model already fits on one device, where any partitioning adds communication for nothing.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Tensor Parallelism — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path