Model Size — AI Glossary

Reference details

Related terms

• Parameter Count
• Quantization
• GGUF

How it works

1. The weight footprint is parameter count multiplied by bytes per parameter, so the same model occupies different amounts depending on the precision it is stored in.
2. Quantisation reduces the bytes per parameter and is where most of the available reduction lives, with the quality cost rising as the bit width falls and not uniformly across layers.
3. Quantised containers usually keep some tensors at higher precision — embeddings and the output head are common exceptions — so the real file size sits above the naive bits-times-parameters figure.
4. Serving memory is weights plus the key-value cache plus activations plus runtime overhead, and only the first of those is what people mean by model size.
5. The cache grows with layers, model width, sequence length and concurrent sequences, which makes it the term that scales with usage rather than with the model.
6. So the question 'does this fit' depends on context length and concurrency, and a model that loads successfully can still fail once it is actually serving requests.

Example: Why the cache can exceed the quantised weights

Take a 7B model with 32 layers and a width of 4,096. At 4-bit the weights are about 7e9 x 0.5 = 3.5e9 bytes, 3.5GB — the figure usually quoted as the model's size. Now the key-value cache at a 8,192-token context, holding a key and a value vector per layer in 16-bit: 2 x 2 x 32 x 4,096 x 8,192 = 4,294,967,296 bytes, about 4.3GB. The cache for one sequence is larger than the entire quantised model. That is why a 3.5GB model does not fit in 6GB of accelerator memory at a long context, and why the first thing to change when it does not fit is the context length rather than the quantisation: halving the context to 4,096 halves the cache to 2.1GB, recovering more than any further weight compression could. Two concurrent sequences at the full 8,192 need 2 x 4.3 = 8.6GB of cache against 3.5GB of weights, so at any real concurrency the cache dominates the budget entirely — and a cache-reduction technique such as grouped-key attention or cache quantisation is worth more than another bit off the weights.

Commonly confused with

Model size and parameter count are different numbers and the first is not a property of the model alone. Parameter count is architectural and fixed; size depends on the storage precision, so one model is 14GB at 16-bit and 3.5GB at 4-bit without being a different model. The more consequential confusion is between model size and memory required to serve, which adds the cache, activations and runtime overhead — the second can be several times the first at long contexts, which is why 'it fits in VRAM' judged on the weight file is a claim that fails under load rather than at load time.

When to use it

Reach for it when:

• Checking whether a model can be loaded on given hardware, which is the first and easiest of the two questions.
• Comparing quantisation variants, where size against measured quality is the trade being made.
• Estimating download and storage requirements for local or edge deployment.
• Alongside a cache calculation, which is what turns a loading question into a serving one.

Reach for something else when:

• As the memory requirement for serving, which adds a cache that can exceed the weights at long contexts.
• As a proxy for capability. Size follows parameters and precision, neither of which orders models by quality.
• Compared across precisions without saying which, since the same model differs severalfold.
• Taken from bits times parameters, since quantised containers keep some tensors at higher precision.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Model Size — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path