Residual Connection — AI Glossary

Reference details

Related terms

• Layer Normalization
• Transformer
• Feed-Forward Network (FFN)

How it works

1. Each sub-layer's output is added to its input rather than replacing it, so the sub-layer learns a modification instead of a transformation.
2. That addition creates an identity path from input to output, which a gradient can traverse without passing through the sub-layer's weights.
3. Gradients therefore reach early layers at usable magnitude, which is what makes very deep stacks trainable at all.
4. The accumulated additions form a residual stream that every layer both reads from and writes into.
5. That framing is why interpretability work treats the stream as a shared communication channel rather than as a sequence of transformations.

Example: Why depth stopped being a barrier

Without the identity path, a gradient reaching layer 1 of an 80-layer stack passes through 80 weight matrices, and if each contributes a factor averaging 0.9 the signal arrives at 0.9⁸⁰ ≈ 0.0002 — effectively zero, so the early layers never learn. The additive path contributes a factor of 1 at every layer, so a route exists whose magnitude does not decay with depth. One architectural change, and the practical ceiling on depth moved by orders of magnitude.

Commonly confused with

Discussed together with normalisation and often merged with it, though they solve different problems. The residual connection provides the identity path that keeps gradients alive through depth; normalisation controls activation scale so each sub-layer sees well-conditioned inputs. Remove the residual and a deep stack will not train; remove normalisation and it trains unstably. Their interaction is what the pre-norm versus post-norm question is about — whether the normalisation sits inside the residual path or outside it.

When to use it

Reach for it when:

• Explaining why transformers can be hundreds of layers deep.
• Reasoning about the residual stream as a shared channel, as interpretability work does.
• Distinguishing gradient flow from activation conditioning.

Reach for something else when:

• As a synonym for normalisation, which addresses a different failure.
• Explaining activation scale or training stability, which normalisation handles.
• Describing attention's routing role, which is unrelated.

Primary source

• He et al., Deep Residual Learning for Image Recognition (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Residual Connection — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path