Data Parallelism — AI Glossary

Reference details

Related terms

• FSDP (Fully Sharded Data Parallel)
• Model Parallelism
• Pipeline Parallelism

How it works

1. Every device holds a complete replica of the model, and the global batch is split so that each device processes a different shard of the data.
2. Each replica computes gradients on its shard independently, which is the embarrassingly parallel part and the reason the strategy scales so easily.
3. Gradients are then averaged across all devices by an all-reduce, so every replica applies the same update and the replicas stay identical — the synchronisation that makes the result equivalent to one large batch.
4. Ring-based all-reduce is the standard implementation, moving each gradient element around a logical ring so that the data each device transmits is proportional to the model size and independent of the device count.
5. Communication is overlapped with the backward pass by reducing each layer's gradients as soon as they are available, which hides much of the cost behind computation that is still running.
6. The strategy requires the whole model, its gradients and its optimiser state to fit on one device; when they do not, the state must be sharded or the model itself split, which is a different strategy.

Example: What the all-reduce costs, and why overlap is not optional

A 7B model produces gradients of the same size as its parameters: at 16 bits that is 7e9 x 2 = 1.4e10 bytes, 14GB, every step. A ring all-reduce moves roughly twice the payload per device — a reduce-scatter pass and an all-gather pass — so about 2 x 14 = 28GB crosses each device's link per step. On a 100 GB/s interconnect that is 28 / 100 = 0.28 seconds of communication. If the step's computation takes 0.5 seconds, the naive total is 0.5 + 0.28 = 0.78 seconds and communication is 0.28 / 0.78 ≈ 36% of every step — more than a third of the cluster's time spent moving numbers rather than computing them. This is why gradient reduction is overlapped with the backward pass rather than run after it: each layer's gradients are reduced while earlier layers are still computing, which hides most of the 0.28 seconds behind work that was happening anyway. It is also why interconnect bandwidth, not accelerator count, is usually what limits data-parallel scaling: at 25 GB/s the same all-reduce takes 28 / 25 = 1.12 seconds and communication becomes 1.12 / 1.62 ≈ 69% of the step, so adding devices buys very little.

Commonly confused with

Data parallelism replicates the model and splits the data; model and tensor parallelism split the model itself. The distinction decides what each one is for: data parallelism is the answer to wanting more throughput on a model that already fits, and it does nothing for a model that does not fit, since every device needs a full copy. Sharded data parallelism sits between them, splitting the optimiser state and gradients across devices while keeping the data-parallel structure — which is how a model too large for one device's full training state is trained without moving to true model parallelism.

When to use it

Reach for it when:

• Models that fit on one device along with their gradients and optimiser state, which is the strategy's precondition.
• Scaling throughput, where more devices processing more data is the simplest available lever.
• Clusters with high-bandwidth interconnect, since the all-reduce cost is the limiting term.
• As the default starting point, with other strategies composed on top when it stops being sufficient.

Reach for something else when:

• Models too large for a single device, where every replica needing a full copy is disqualifying.
• Low-bandwidth interconnect, where communication can exceed a third of each step and extra devices buy little.
• Without overlapping reduction with the backward pass, which leaves the full communication cost exposed.
• At very large device counts without a sharded variant, where per-device optimiser state becomes pure duplication.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Data Parallelism — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path