Feed-Forward Network (FFN) — AI Glossary

Reference details

Related terms

• Transformer
• Residual Connection
• Layer Normalization

How it works

1. Each transformer block contains a two-layer network applied independently at every token position, with no mixing between positions.
2. The first projection expands the hidden dimension — conventionally about fourfold — and the second projects back down.
3. A non-linearity sits between them, and gated variants split the expansion into two projections multiplied together, which is why they are wider in parameter count for the same nominal ratio.
4. Because attention mixes across positions and this does not, the two sub-layers have distinct roles: attention moves information between tokens, this transforms it in place.
5. It holds most of the block's parameters, which is why interpretability work looking for stored factual associations looks here rather than at attention.

Example: Where the parameters actually are

At a hidden dimension of 4,096, attention's four projections hold roughly 4 x 4,096² ≈ 67M parameters per layer. The feed-forward network at a fourfold expansion holds 2 x 4,096 x 16,384 ≈ 134M — about twice attention, so roughly two-thirds of the block. A gated variant with three matrices at that width pushes it higher still. This is why parameter count scales with the expansion ratio more than with head count, and why quantising or pruning this sub-layer moves memory most.

Commonly confused with

Sometimes assumed to be the whole transformer block, when it is one of two sub-layers and the one that does not communicate between positions. Attention routes information across the sequence; this transforms each position independently, so removing it would leave a model that could route but not compute. It is also not a mixture-of-experts layer, though MoE replaces it: an MoE block swaps this dense network for several of them with a router, which is why MoE parameter counts balloon while compute per token does not.

When to use it

Reach for it when:

• Explaining where a transformer's parameters and stored associations sit.
• Reasoning about why expansion ratio drives model size more than head count.
• Understanding what mixture-of-experts replaces and why its parameter count grows.

Reach for something else when:

• Explaining how information moves between tokens, which is attention's role.
• As a synonym for the transformer block, which contains both sub-layers.
• Describing positional behaviour, which this sub-layer is indifferent to.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Feed-Forward Network (FFN) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path