Layer Normalization — AI Glossary

Reference details

Related terms

• Residual Connection
• Transformer
• Feed-Forward Network (FFN)

How it works

1. Activations are normalised across the hidden dimension for each token independently, so the statistics come from that token's own features.
2. That independence is the point: unlike batch normalisation there is no dependence on other examples, so behaviour at batch size one matches training.
3. Learned scale and shift parameters follow, letting the network undo the normalisation where it is unhelpful.
4. Placement matters more than the operation: normalising before each sub-layer leaves the residual path unnormalised, which keeps gradients well-behaved in deep stacks.
5. Variants drop the mean-centring and normalise by root-mean-square alone, which is cheaper and empirically as good in transformers.

Example: Why placement decides whether a deep stack trains

With normalisation after each sub-layer, the residual stream is renormalised at every one of 80 layers, so the identity path is repeatedly rescaled and gradient magnitude at the bottom depends on all 80 scalings. Moving it before the sub-layer leaves the residual path clean from input to output, so a gradient reaches layer 1 through an unmodified identity route. The operation is identical in both; the training stability of very deep models comes from where it sits.

Commonly confused with

Confused with batch normalisation, and the difference is which axis supplies the statistics. Batch normalisation uses the batch, so inference must use running averages and behaviour depends on batch composition; this uses a single token's own features, so a batch of one behaves exactly as training did. That is why transformers use it and convolutional vision models historically did not. It is also not the same as the residual connection it is usually discussed beside — one normalises, the other provides the identity path, and they solve different depth problems.

When to use it

Reach for it when:

• Explaining why transformers train stably at depth and why pre-norm became standard.
• Reasoning about why batch size does not change a transformer's per-example behaviour.
• Distinguishing the normalisation operation from where it is placed.

Reach for something else when:

• As a synonym for normalisation generally, since the axis is the defining property.
• Explaining gradient flow through depth, which is the residual connection's role.
• Describing the RMS variant as identical, since it drops mean-centring.

Primary source

• Ba et al., Layer Normalization (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Layer Normalization — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path