Data Deduplication — AI Glossary

Reference details

Related terms

• Common Crawl
• Pretraining Data
• Benchmark Contamination

How it works

1. Each document is reduced to a set of overlapping shingles, and a signature is built by hashing those shingles under many independent permutations and keeping the minimum under each.
2. The probability that two documents share a given minimum equals their Jaccard similarity, so comparing signatures estimates similarity without comparing documents.
3. The signature is then split into bands of several rows each, and two documents become candidates if they match exactly on any one band.
4. Banding converts an all-pairs problem into a hash lookup, which is the only reason the method runs at crawl scale — nothing is compared until it has already collided.
5. Band and row counts set a soft threshold rather than a sharp one: with 32 bands of 4 rows, a pair at Jaccard 0.67 collides with probability 1 - (1 - 0.67^4)^32 ≈ 99.9%, while a pair at 0.3 collides with probability ≈ 22.9%.
6. Those two numbers are the dial: raising rows per band sharpens the cutoff and raising band count lowers it, so the parameters are chosen to place the transition where the corpus needs it.

Example: Why exact matching finds almost nothing

Two copies of the same article, one with a cookie banner and a different byline. Shingle each into overlapping word windows: say 100 shingles apiece, 80 shared. Jaccard similarity is 80 / (100 + 100 − 80) = 80/120 ≈ 0.67 — plainly the same document. An exact-hash pass sees two different strings and keeps both. MinHash estimates that ratio cheaply enough to run over billions of documents, which is the only reason near-duplicate removal is tractable at pretraining scale.

Commonly confused with

Training deduplication and evaluation decontamination use the same machinery for different goals. Deduplication improves the corpus; decontamination removes documents overlapping the test set so the reported score means something. A model can be well-deduplicated and still contaminated if nobody checked against the benchmark.

When to use it

Reach for it when:

• Any pretraining or continued-pretraining corpus assembled from web crawl
• Before reporting benchmark numbers, as a decontamination pass against the eval sets
• RAG corpora — near-duplicate chunks crowd out diverse evidence in the top-k

Reach for something else when:

• Where repetition is signal — legal boilerplate or code idioms may legitimately recur
• Aggressive thresholds on small domain corpora, which can strip out genuine coverage
• As a substitute for quality filtering: deduplicated junk is still junk

Primary source

• Lee et al., Deduplicating Training Data Makes Language Models Better (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Data Deduplication — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path