Model Parallelism — AI Glossary

Reference details

Related terms

• Distributed Inference
• Pipeline Parallelism
• DeepSpeed

How it works

1. Tensor parallelism splits the feed-forward block in a deliberate pairing: the first matrix is divided by columns so each device computes a slice of the wider hidden activation with no communication at all.
2. The second matrix is then divided by rows, which means each device produces a partial sum of the full output and one all-reduce is required to add them together.
3. Attention splits along heads, which partitions cleanly because heads are independent until their outputs are concatenated and projected — giving the second collective in the block.
4. Two collectives per transformer block is therefore the structural cost, and with tens of blocks a forward pass contains dozens of synchronisation points that every device must reach.
5. Pipeline parallelism avoids that entirely by assigning whole layers to devices and passing activations forward point to point, so traffic is one tensor per stage boundary rather than a collective per block.
6. What it buys in bandwidth it pays in idleness, because a stage cannot start until the one before it has produced something — which is why the two are combined rather than chosen between at scale.

Example: Two ways to split, with different network bills

Tensor parallelism splits individual weight matrices across devices, so every device holds a slice of every layer and they must synchronise within each layer — with roughly two collective operations per transformer block, a deep model reaches hundreds of synchronisation points per forward pass. That demands very fast links, which is why it is normally kept inside one node. Pipeline parallelism instead assigns whole layers to different devices and passes activations forward point-to-point: far less traffic, tolerant of slower links, but it introduces idle time while later stages wait for earlier ones. Large runs combine both, plus data parallelism, precisely because they fail in different directions.

Commonly confused with

Model parallelism splits the model because it does not fit; data parallelism replicates the model and splits the batch. Sharded strategies like FSDP and ZeRO look like model parallelism but are data-parallel underneath — they shard storage and reconstruct full layers on demand, rather than permanently distributing the computation.

When to use it

Reach for it when:

• Models whose weights exceed a single device, where sharding is the only option
• Tensor parallelism within a node on fast interconnect; pipeline parallelism across nodes
• Serving very large models, where the alternative is not serving them

Reach for something else when:

• Models that fit on one device — the communication is pure overhead
• Tensor parallelism across slow links, where synchronisation dominates
• As a first response to memory pressure: quantization and KV-cache limits are cheaper

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Model Parallelism — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path