Parameter Count — AI Glossary

Reference details

Related terms

• Model Size
• Model Architecture
• Scaling Law

How it works

1. Almost all parameters sit in two places per transformer block: the attention projections and the feed-forward network, with layer norms contributing a negligible count.
2. The four attention projections cost about four times the square of the model width, and a feed-forward network expanding to four times the width costs about eight times it — so a block is roughly twelve times the width squared.
3. That makes the feed-forward network about 8/12 ≈ 66.7% of every block, which is why mixture-of-experts replaces the feed-forward layer specifically rather than attention.
4. For a width of 4,096 across 32 blocks that is 12 x 4,096^2 x 32 = 6,442,450,944 parameters in the stack.
5. The embedding table adds vocabulary times width — 128,256 x 4,096 = 525,336,576 — bringing the total to 6,967,787,520, which is the arithmetic behind a model being called 7B.
6. Because the embedding term scales with vocabulary rather than depth, it is a small fraction of a large model and a substantial one of a small model, which is why very small models often tie their input and output embeddings.

Example: Total, active, and what actually has to fit

Weights alone at fp16 cost 2 bytes per parameter: a 7B model is 7e9 × 2 = 14 GB before any KV cache. Quantised to 4 bits it is roughly 3.5 GB, which is why a laptop can run it. Mixture-of-experts breaks the link between size and cost in the other direction: Mixtral 8x7B holds about 46.7B parameters in total but routes each token through roughly 12.9B of them, so it needs the memory of a 47B model and the compute of a 13B one. One number for such a model tells you almost nothing.

Commonly confused with

Parameter count is a capacity proxy, not a quality measure. Training tokens, data quality and post-training often matter more, which is why smaller recent models beat larger older ones. For MoE, always ask whether a figure is total or active parameters — they answer different questions, memory and compute respectively.

When to use it

Reach for it when:

• Sizing hardware: memory footprint follows total parameters and precision directly
• Comparing models within one family and generation, where other variables are held still
• Estimating quantization headroom before committing to a deployment target

Reach for something else when:

• Predicting quality across families or generations — the correlation is weak
• Comparing dense and MoE models on a single number
• Cost modelling for hosted APIs, where you pay per token and never see the parameters

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Parameter Count — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path