Pretraining — AI Glossary

Reference details

Related terms

• Foundation Model
• Fine-Tuning
• Supervised Fine-Tuning (SFT)

How it works

1. A corpus is assembled and filtered at scale, and the filtering decisions shape the model as much as the architecture does.
2. Text is tokenised once into a fixed vocabulary, which the model is then bound to for its whole life.
3. Training optimises next-token prediction across the corpus, usually for roughly one pass, since repeating data gives diminishing returns and risks memorisation.
4. Compute scales as roughly parameters times tokens, which is the budget both scaling laws and procurement are expressed in.
5. The output is a base model: general capability, and no notion that a prompt is a request — that arrives in later tuning stages.

Example: Why the run is a procurement decision

Training compute is approximately 6 x parameters x tokens FLOPs. A 7B model on 2T tokens is 6 x 7e9 x 2e12 = 8.4e22 FLOPs. An accelerator delivering 4e14 FLOPs per second at 40% utilisation does 1.6e14 effective, so the run needs about 5.25e8 seconds of single-device time — roughly 17 device-years, or about 1,000 devices for six days. That arithmetic, not the code, is why pretraining is concentrated among organisations that can buy a cluster.

Commonly confused with

Pretraining builds general capability from scratch on web-scale data and costs a training run few organisations can fund. Fine-tuning adapts a model that already exists. Continued pretraining sits between them — more next-token training on domain data, without task-specific pairs — and is the right label for the work often mislabelled as fine-tuning when the aim is absorbing a domain rather than learning a task format.

When to use it

Reach for it when:

• Explaining where a model's general capability and its knowledge cutoff come from.
• Reasoning about compute budgets, scaling ratios, or why a tokeniser cannot be changed later.
• Distinguishing capability that must be trained in from behaviour that can be tuned or prompted.

Reach for something else when:

• Describing adaptation of an existing model, which is fine-tuning or continued pretraining.
• Any realistic plan for adding domain knowledge, where retrieval or tuning is the lever.
• Explaining refusal behaviour or instruction-following, which pretraining does not produce.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Pretraining — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path