Megatron-LM — AI Glossary

Reference details

Related terms

• DeepSpeed
• FSDP (Fully Sharded Data Parallel)
• Model Parallelism

How it works

1. Tensor parallelism splits each weight matrix across devices, so a single layer's matrix multiply is performed in pieces and the partial results are summed by a collective.
2. That summation happens twice per transformer block, which is why the tensor-parallel group is kept inside one node where interconnect bandwidth is highest.
3. Pipeline parallelism assigns consecutive layers to different device groups, so a batch flows through stages and each stage holds only its own slice of the weights.
4. Pipelines idle at the start and end of every batch, so the batch is split into microbatches that keep stages fed, and the microbatch count sets how much of that idling remains.
5. Data parallelism then replicates the whole model-parallel group and averages gradients across replicas, which is the dimension that scales throughput once the model already fits.
6. The three degrees multiply to the device count, so a cluster is configured by factorising it — and the factorisation, not the total, determines what the hardware achieves.

Example: The pipeline bubble is set by microbatches, not by hardware

A 530B-parameter model cannot be held by one device: at fp16 its weights alone are about 1,060 GB, so 1,060 / 80 = 13.25 and at least 14 80 GB devices are needed before any optimiser state exists. Add mixed-precision Adam at roughly 16 bytes per parameter and the trainable state is 530e9 x 16 = 8,480 GB, which needs 8,480 / 80 = 106 devices for state alone. Model parallelism is not an optimisation here, it is the precondition for the job existing. Configuring 1,024 devices means factorising them, and 8 x 16 x 8 = 1,024 assigns tensor-parallel 8 inside each node, pipeline-parallel 16 across the layer stack, and data-parallel 8 across replicas. The pipeline degree then carries a cost that has nothing to do with the hardware. With P stages and M microbatches the fraction of time stages sit idle is (P - 1) / (M + P - 1), so at P = 16 and M = 16 that is 15 / 31 ≈ 48.4% — nearly half the cluster idle on a correctly configured job. Raising the microbatch count to 128 gives 15 / 143 ≈ 10.5%. Nothing about the devices changed; the same 1,024 devices went from roughly half-idle to nine tenths busy because one scheduling parameter moved, which is why microbatch count is the first thing to check on a pipeline-parallel run that is slower than its FLOPS suggest.

Commonly confused with

Megatron-LM is not an alternative to DeepSpeed or FSDP so much as a solution to a different part of the problem, and the two are routinely combined. FSDP and ZeRO shard optimiser state, gradients and parameters across data-parallel ranks, which reduces memory without splitting any individual operation. Megatron splits the operations themselves, which is what allows a single layer too large for one device to be computed at all. A frontier training run typically uses Megatron's tensor and pipeline parallelism for the model dimension and ZeRO-style sharding across the data dimension. The second confusion is treating it as a serving framework: its parallelism strategies are designed around training's backward pass and gradient collectives, and inference stacks make different trade-offs for the same model.

When to use it

Reach for it when:

• When a single layer's weights exceed one device's memory, which tensor parallelism exists to address and sharding does not.
• For training runs whose optimiser state alone needs many devices, as the worked case needed 106 for state.
• Alongside ZeRO-style sharding on the data dimension, since the two address different memory terms.
• When tuning throughput on a pipeline-parallel job, where the microbatch count moved idle time from about 48.4% to 10.5%.

Reach for something else when:

• For models that fit one device, where data parallelism alone is simpler and faster.
• For inference serving, whose parallelism trade-offs differ from training's.
• With tensor-parallel groups spanning nodes, since the per-block collectives then cross the slowest link in the system.
• As a substitute for sharding, which reduces a memory term that splitting operations does not touch.

Primary source

• Shoeybi et al., Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Megatron-LM — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path