ModelRefs / Mixture of Experts (MoE) — AI Glossary

Mixture of Experts (MoE) — AI Glossary

A neural architecture routing each token to a small subset of expert sub-networks, decoupling parameter count from active compute.

Overview

MoE models (Mixtral 8x7B, GPT-4 estimated, Gemini 1.5) replace each FFN layer with N expert networks and a gating router that selects top-k experts per token. Active parameters per forward pass ≈ dense_params / N, enabling 3–8× parameter scale-up without proportional compute increase. Training challenges: load balancing across experts, router collapse.

Reference details

Topic	architecture
Also known as	MoE, sparse MoE
Last reviewed	2026-06-24

How it works

A small linear layer — the router — scores every expert for each token, and the top-scoring few are selected for that token alone rather than for the sequence.
The chosen experts' outputs are combined as a weighted sum using the router's own scores, renormalised across just the selected experts, so the gate is part of the differentiable path.
Routing is per token and per layer, so a sequence's tokens take different paths through the network and one token may use entirely different experts at layer three and layer thirty.
Left alone the router collapses onto a few experts, since an expert that is chosen more improves faster and is then chosen more still — so an auxiliary loss penalises uneven assignment.
A capacity factor caps how many tokens any one expert may accept per batch, and tokens arriving beyond that cap are dropped past the expert rather than queued — they pass through on the residual connection unprocessed.
That dropping is invisible in the output and shows up only as degraded quality under skewed batches, which is why capacity and balance are monitored during training rather than assumed.

Example: Sparse in compute, dense in memory

Take eight experts with the top two routed per token. Each token runs through 2 / 8 = 25% of the expert parameters, so the arithmetic per token is close to a model a quarter the size. The memory is not sparse at all: any expert may be selected by the next token, so all eight must be resident. You get the compute bill of a small model and the VRAM bill of a large one. That single asymmetry explains most practical MoE decisions — why they are attractive to serve at scale and awkward to run on one accelerator, and why quoting a single parameter count for them is meaningless.

Commonly confused with

Total and active parameters are two different numbers and neither substitutes for the other: total predicts memory, active predicts compute and latency. An MoE is also not an ensemble — an ensemble runs several models and combines outputs, while a router here picks a subset of sub-networks *inside* one forward pass, and only those run.

When to use it

Reach for it when:

Serving at scale, where the compute saving repays the memory cost across many requests
Reasoning about why a very large model serves faster than its parameter count suggests
Reading model cards, where total and active counts must both be stated to mean anything

Reach for something else when:

Memory-constrained deployment — a dense model of the active size is the fair comparison
Assuming quality tracks total parameters; the active path does the work
Fine-tuning without accounting for routing, which can collapse onto a few experts

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Mixture of Experts (MoE) — AI Glossary.

Frequently asked questions

What is Mixture of Experts (MoE)?

A neural architecture routing each token to a small subset of expert sub-networks, decoupling parameter count from active compute.

Is Mixture of Experts (MoE) the same as MoE?

Yes — MoE, sparse MoE are common aliases for Mixture of Experts (MoE).

What concepts are related to Mixture of Experts (MoE)?

Closely related concepts include feed forward network, model architecture, parameter count.