Chinchilla Scaling — AI Glossary

Reference details

Related terms

• Scaling Law
• Neural Scaling
• Pretraining Data

How it works

1. Fix a compute budget. Training compute scales roughly as parameters × tokens, so within that budget the two can be traded against each other.
2. Train many models across the parameter/token frontier for the same budget and record final loss for each, rather than assuming the relationship.
3. Fit loss as a function of both terms. The fitted surface has a minimum that is interior, not at the large-model edge where practice had been sitting.
4. Read the ratio off the minimum: compute-optimal training puts roughly 20 tokens through each parameter, so a budget that bought a very large model trained briefly instead buys a smaller one trained far longer.
5. The result is about training efficiency only. Inference cost falls out of parameter count, which is why the finding pushed deployed models smaller for reasons the paper did not need to argue.

Example: The same compute, spent two ways

Take parameters × tokens as the budget proxy. A 280B-parameter model trained on 300B tokens gives 280 × 300 = 84,000 units. A 70B model trained on 1.4T tokens gives 70 × 1400 = 98,000 — the same order, with the second trained on well over four times the data at a quarter of the parameters. At serving time the 70B model is also the cheaper one to run, so the smaller-and-longer configuration wins twice. The arithmetic is the whole argument, which is why it redirected the field so quickly.

Commonly confused with

Treated as a rule that 20 tokens per parameter is correct, which is narrower than it sounds. It is the compute-optimal ratio for minimising training loss at a fixed training budget, and it says nothing about the budget that includes serving. If a model will answer billions of requests, pushing well past 20 tokens per parameter buys a smaller model that is cheaper for its whole life — which is why production models are now routinely trained far beyond the Chinchilla point and are not thereby violating it. It also supersedes rather than contradicts the earlier scaling work: same method, better-fitted curves, different minimum.

When to use it

Reach for it when:

• Allocating a training budget between model size and dataset size.
• Explaining why a smaller model trained longer can match a much larger one.
• Arguing for total cost of ownership, including inference, rather than training compute alone.

Reach for something else when:

• As a target for a model whose dominant cost is serving — that optimum sits past the Chinchilla ratio.
• As a claim about capability per parameter in general; it is a statement about loss at fixed training compute.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Chinchilla Scaling — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path