Embedding Dimension — AI Glossary

Reference details

Related terms

• Parameter Count
• Multi-Head Attention (MHA)
• Vocabulary Size

How it works

1. Every token is mapped to a vector of this width, and that width is the size of the residual stream carried through all layers. Each sub-layer reads from it and writes back to it at the same width.
2. Attention splits the width across heads, so each head operates on dimension equal to the model width divided by the head count. Changing the width without changing the head count changes what each head has to work with.
3. Parameter cost grows with the square of the width in the dense layers, since each projection maps width to width or to a fixed multiple of it. Doubling the width roughly quadruples those parameters.
4. The embedding and output matrices grow linearly in both width and vocabulary size, which makes them a large fixed cost for multilingual models with big vocabularies.
5. Activation memory and KV-cache size grow linearly in width, so width is also a serving-cost decision and not only a training one.
6. It is fixed at architecture time. Every weight matrix is shaped by it, so it cannot be changed by fine-tuning — unlike a sampling parameter or even a context-length extension.

Example: Why doubling the width is not a 2x decision

A transformer layer's dense parameters scale with the square of the width: the attention projections are four matrices of width-by-width, giving 4d^2, and a feed-forward block expanding by 4x is two matrices of width-by-4-width, giving 8d^2 — about 12d^2 per layer in total. At d = 4,096 that is 12 x 4,096^2 = 12 x 16,777,216 = 201,326,592, roughly 201M parameters per layer. Double the width to 8,192 and it becomes 12 x 8,192^2 = 12 x 67,108,864 = 805,306,368, about 805M — four times as many, from a 2x change. Across 32 layers that is 32 x 0.201 = 6.4B against 32 x 0.805 = 25.8B. Meanwhile the KV cache per token grows only linearly, since it stores two vectors per layer: at 2 bytes each, 2 x 2 x 32 x 4,096 = 524,288 bytes per token against 2 x 2 x 32 x 8,192 = 1,048,576 — exactly 2x. So width buys capacity at a quadratic price in parameters and a linear price in serving memory, which is why depth, head count and feed-forward ratio are tuned around it rather than width simply being increased.

Commonly confused with

Model width and embedding-model output dimension are different numbers that share a name. The width discussed here is internal to a transformer — the residual stream every layer reads and writes. An embedding model's dimension is the size of the vector it emits for an entire text, which is what a vector database stores and what determines index size. They are unrelated in value: a model with a 4,096-wide residual stream may emit 1,024-dimensional text embeddings, because the output head projects down. Confusing them leads to sizing a vector index against the wrong number by a factor of several.

When to use it

Reach for it when:

• Reading an architecture specification, where width together with depth and head count describes the shape of the model.
• Estimating parameter count or memory from a configuration, where the quadratic-versus-linear distinction is what makes the estimate right.
• Sizing a KV cache for a serving deployment, which scales linearly in width, layers and sequence length.
• Comparing models of similar parameter count, where a wide-shallow and a narrow-deep model behave differently despite the same total.

Reach for something else when:

• As a proxy for capability. Width is one term among several, and parameter count alone does not order models by quality.
• Confused with an embedding model's output dimension, which is the number that sizes a vector index.
• As something adjustable after training. Every weight is shaped by it; changing it means a new model.
• Scaled without revisiting head count and feed-forward ratio, since each head's working dimension is derived from it.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Embedding Dimension — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path