ModelRefs / DeepSpeed — AI Glossary

DeepSpeed — AI Glossary

Microsoft's distributed training library implementing ZeRO optimizer stages, pipeline parallelism, and mixed precision to train 100B+ parameter models.

Overview

DeepSpeed ZeRO (Zero Redundancy Optimizer) shards optimizer states, gradients, and parameters across data-parallel workers, eliminating redundant memory. ZeRO-3 shards all three, enabling single-GPU-equivalent memory per device. Used to train BLOOM, Falcon, and Megatron-class models. Integrates with PyTorch DDP and Hugging Face Accelerate.

Reference details

Topic	training
Last reviewed	2026-06-24

How it works

Data-parallel training normally replicates optimiser state, gradients and parameters on every worker, so the same bytes are held as many times as there are workers.
Stage one shards the optimiser state across workers, which removes the largest of the three without changing how the forward or backward pass runs.
Stage two additionally shards gradients, so each worker reduces and retains only the slice it owns rather than a full gradient copy.
Stage three shards the parameters themselves, which changes execution: a layer's weights are gathered from the workers holding them immediately before use and released immediately after.
That gather is the trade — memory falls roughly in proportion to the worker count while communication rises, so stage three needs interconnect bandwidth to be worth choosing over stage two.
Offload moves shards further out, to host memory or to disk, buying capacity at a bandwidth an order of magnitude lower — which makes a model trainable rather than making it train quickly.

Example: The memory arithmetic ZeRO attacks

Mixed-precision training with Adam holds roughly 16 bytes per parameter: 2 for the fp16 weights, 2 for the fp16 gradients, and 12 for the fp32 master copy plus the two optimizer moments. A 7-billion-parameter model therefore needs about 7e9 × 16 = 112 GB for states alone — before activations, and well beyond a single 80 GB accelerator. ZeRO stage 3 shards all three categories across data-parallel workers, so across 8 GPUs each holds roughly 112 / 8 = 14 GB. The model did not shrink; the redundancy did.

Commonly confused with

DeepSpeed is a library; ZeRO is the sharding technique it introduced. They are not interchangeable — PyTorch FSDP implements the same idea natively, so you can shard without DeepSpeed, and DeepSpeed offers pipeline parallelism and inference kernels that are not ZeRO.

When to use it

Reach for it when:

Training or full fine-tuning where optimizer state, not the weights, is what exceeds memory
Multi-node runs needing pipeline and tensor parallelism alongside sharding
Existing Hugging Face training code, which integrates it through a config file

Reach for something else when:

Parameter-efficient fine-tuning — LoRA sidesteps the optimizer-state problem entirely
Inference serving, where a dedicated engine is the better fit
Single-GPU work: sharding across one worker adds complexity and buys nothing

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to DeepSpeed — AI Glossary.

Frequently asked questions

What is DeepSpeed?