FSDP (Fully Sharded Data Parallel) — AI Glossary

Reference details

Related terms

• DeepSpeed
• ZeRO Optimizer
• Data Parallelism

How it works

1. The model is wrapped into units, conventionally one per transformer block, and each unit's parameters are flattened into a single contiguous buffer before being split across workers.
2. Flattening matters because collectives operate on one large buffer far more efficiently than on hundreds of small tensors, so the wrapping granularity is a performance decision rather than a bookkeeping one.
3. A forward pre-hook on each unit all-gathers its full parameters from the workers holding them, the unit computes, and a post-hook frees the gathered copy immediately.
4. The same sequence runs in reverse during the backward pass, so each unit's parameters are gathered twice per step and the collective count scales with the number of units.
5. Unit granularity therefore trades two costs directly: larger units mean fewer, bigger collectives and a higher peak because more parameters are resident at once.
6. Gradients are reduce-scattered rather than all-reduced, so each worker receives only the slice it owns and updates only its shard of the optimiser state — which is where the memory saving ultimately lands.

Example: The memory you save is bought with network traffic

Sharding puts 1/N of the parameters on each of N workers, which is what makes a model too large for one device trainable. But the maths still needs whole layers: before each layer's forward pass the workers all-gather its full parameters, use them, then discard the copy and re-shard. Every layer, every step, in both directions. That is why sharded training is sensitive to interconnect in a way that plain data parallelism is not — the same job that runs well on fast intra-node links can become communication-bound across slower ones, with the GPUs idle waiting.

Commonly confused with

FSDP is PyTorch's native implementation of the same idea as DeepSpeed ZeRO stage 3, not a competing technique. Choosing between them is mostly an ecosystem and tooling decision. Neither is a substitute for parameter-efficient fine-tuning: if the optimizer state is the problem and full-model quality is not required, LoRA removes the problem instead of distributing it.

When to use it

Reach for it when:

• Full fine-tuning or pretraining where optimizer state and gradients exceed device memory
• PyTorch-native stacks, where it avoids an extra dependency and config surface
• Multi-node runs on fast interconnect, where the all-gather traffic is affordable

Reach for something else when:

• Anything LoRA or another PEFT method can do — far less machinery for the same outcome
• Slow interconnects, where communication dominates and utilisation collapses
• Inference, which has different sharding needs and dedicated engines

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to FSDP (Fully Sharded Data Parallel) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path