Pretraining Data — AI Glossary

Reference details

Related terms

• Common Crawl
• Data Deduplication
• Chinchilla Scaling

How it works

1. Raw crawl data is first converted from markup to text, which is itself lossy and consequential — boilerplate, navigation and template text removed badly becomes a large fraction of what the model sees.
2. Language identification and quality filtering follow, the latter usually a classifier trained to recognise text resembling curated reference sources rather than a hand-written rule set.
3. Deduplication runs at both document and substring level, and it is the step with the largest measured effect on final quality: repeated text is memorised rather than generalised from.
4. Decontamination then removes items overlapping published benchmark test sets, which is a best-effort operation since it can only cover benchmarks known at training time.
5. Sources are mixed at chosen ratios with high-quality corpora upsampled — repeated more than once per epoch — so the effective composition differs from the raw proportions available.
6. None of these ratios or filters is usually published, which is why coverage of a domain or language has to be inferred from behaviour rather than read off a specification.

Example: Compute-optimal is not deployment-optimal

The Chinchilla result says that for a fixed training budget, roughly 20 tokens per parameter minimises loss — a 7B model would take about 7e9 × 20 = 140 billion tokens. Almost no deployed model follows this. Llama 3's 8B was trained on roughly 15 trillion tokens, about 1,875 tokens per parameter — nearly a hundred times the compute-optimal ratio. The reason is that Chinchilla optimises training cost, while a model serving billions of requests is dominated by inference cost. Over-training a smaller model buys a permanently cheaper model to run.

Commonly confused with

Pretraining data is not fine-tuning data and not a knowledge base. It sets what the model absorbed statistically, not what it can look up; adding a document to a RAG corpus is not adding it to the model. A knowledge cutoff describes the pretraining corpus, not the system's access to current information.

When to use it

Reach for it when:

• Judging a model's likely coverage of a domain, language or code ecosystem
• Reasoning about cutoffs, staleness, and where retrieval is mandatory
• Planning continued pretraining, where composition and deduplication dominate the outcome

Reach for something else when:

• Treating undisclosed composition as if it were known — most frontier recipes are not published
• Assuming presence in pretraining means reliable recall of a specific fact
• Comparing token counts across models as a quality ranking

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Pretraining Data — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path