Common Crawl — AI Glossary

Reference details

Related terms

• Pretraining Data
• Data Deduplication
• Data Augmentation

How it works

1. A crawler fetches web pages at scale on a repeating schedule and publishes the raw responses as archive files.
2. What is published is unfiltered: boilerplate, duplicates, machine translation, markup and low-quality text are all present as crawled.
3. Usable training data requires a pipeline — language identification, boilerplate stripping, quality scoring, deduplication, and removal of personal and unsafe content.
4. Deduplication matters disproportionately, because near-duplicate documents are abundant on the web and repeated text is memorised rather than learned from.
5. Published derived corpora exist precisely because that filtering is expensive and the choices in it shape the resulting model.

Example: Why the filtering is the dataset

Aggressive pipelines commonly discard the large majority of crawled text — order-of-magnitude reductions are typical once language filtering, boilerplate removal and deduplication have run. So the corpus a model actually trains on is a small, heavily curated minority of what was crawled, and two teams starting from the same archive produce materially different datasets. The crawl is a raw input; the filtering decisions are where the training data is actually created.

Commonly confused with

Referred to as a training dataset, which it is not in usable form. It is a raw crawl archive, and every model trained from it used a filtered derivative — so citing it as the data source omits the step that did most of the work. It is also not the web: a crawl is a sample constrained by what was reachable, permitted and scheduled, which skews toward large, open, link-rich sites. Treating it as representative of the web overstates coverage of anything behind authentication or outside the crawl's reach.

When to use it

Reach for it when:

• Explaining where web-scale pretraining data originates and what it costs to prepare.
• Discussing why filtering and deduplication decisions shape a model.
• Reasoning about coverage bias in pretraining corpora.

Reach for something else when:

• As the name of a training dataset, since every real use is a filtered derivative.
• As a proxy for the whole web, which it under-samples in predictable directions.
• Where a curated domain corpus is what the task actually needs.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Common Crawl — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path