Web Corpus Cleaning: Filtering, Deduplication, and Mixing

The Problem

You have a data pipeline. You have terabytes of Common Crawl WET files. You do not have a training corpus yet.

Raw Common Crawl is famously dirty. Independent audits of web-scale crawls consistently find that the majority of documents are unusable for language modeling: HTML boilerplate, navigation menus, cookie banners, SEO keyword farms, machine-generated spam, mojibake, adult content, personally identifiable information, and enormous amounts of exact and near-duplicate text. The OpenAI GPT-3 paper noted that Common Crawl quality was so low that it required a separate classifier-based filter; the Gopher paper (Rae et al. 2021) showed that a well-designed rule filter could remove roughly half of a web crawl while improving downstream perplexity; and the FineWeb report (Penedo et al. 2024) demonstrated that deduplication alone changes the effective token distribution enough to matter for benchmarks.

Quality is not a luxury for small models. A 1B-parameter model trained on clean data can outperform a 3B-parameter model trained on raw crawl. The reason is simple: the smaller model has fewer parameters to waste on memorizing navigation footers and duplicate privacy policies. Cleaning is the highest-leverage engineering step in pre-training, and it is almost entirely deterministic code you can write, test, and version.

This lesson builds the three-stage pipeline that turns a batch of raw web documents into a deduplicated, source-weighted training mix:

graph LR
    A[Raw documents] --> B[Rule filter]
    B --> C[MinHash + LSH]
    C --> D[Temperature mixer]
    D --> E[Training mix]

    style A fill:#1a1a2e,stroke:#e94560,color:#fff
    style B fill:#1a1a2e,stroke:#e94560,color:#fff
    style C fill:#1a1a2e,stroke:#e94560,color:#fff
    style D fill:#1a1a2e,stroke:#e94560,color:#fff
    style E fill:#1a1a2e,stroke:#16c784,color:#fff

The Concept

Why most of Common Crawl is unusable

A monthly Common Crawl snapshot contains hundreds of terabytes of WET text. The usable fraction is small. The failure modes fall into a few categories:

Gopher and C4 approached this differently. Gopher (Rae et al. 2021) published an explicit checklist of heuristics. C4 (Raffel et al. 2019) used a smaller set of aggressive filters tuned for English T5 pre-training. CCNet (Wenzek et al. 2019) added language identification and perplexity filtering. FineWeb (Penedo et al. 2024) combined these ideas with modern deduplication at Common Crawl scale.

Gopher's rule checklist

The Gopher filters are intentionally simple. They do not need a learned model, which means they are fast, auditable, and easy to reproduce. The most important rules are:

1. Minimum document length. Documents below a word-count threshold are almost always stubs, error pages, or navigation.
2. Symbol ratio. Too many non-alphanumeric, non-whitespace characters often means encoding garbage, markup leakage, or source code rendered as text.
3. Capitalization ratio. A high ratio of uppercase words is a strong spam signal.
4. Duplicate line ratio. Pages that repeat the same line many times are usually error messages, terms of service, or generated tables.
5. Duplicate paragraph ratio. Boilerplate footers and headers create repeated paragraphs.
6. Stopword ratio. Natural language contains articles, prepositions, and pronouns. A document with almost no stopwords is probably a keyword list or identifier dump.
7. Ellipsis ending. A document ending in ... is often truncated or placeholder text.
8. Mean word length. Extremely long average word length suggests concatenated identifiers or obfuscated text.

Each rule has a threshold, and every rejected document is counted by rule. That count is the survival funnel: if a rule removes 40% of your data, you need to look at what it is removing before you trust it.

C4 heuristics

C4 uses a smaller but overlapping set of filters. A document is removed if:

• it has fewer than three sentences;
• any line is shorter than three words and ends in punctuation (likely a menu item);
• the word count is below a threshold;
• it contains the word javascript as a line;
• it contains lorem ipsum;
• it contains a curly bracket { (likely source code or JSON);
• any phrase repeats more than once in a document.

C4 is tuned for English. Multilingual pipelines usually replace some of these rules with language-specific heuristics or use CCNet's language-id plus perplexity approach.

From exact to near deduplication

Exact deduplication is trivial: hash every document with SHA-256 and keep one copy per hash. The real problem is near-duplicates. Two copies of the same news article with different advertisements are byte-for-byte different but 95% identical. Exact hashing misses them.

The standard solution is MinHash plus Locality-Sensitive Hashing (LSH).

1. Shingle. Convert each document into a set of k-grams. A 5-word shingle of "the quick brown fox jumps" is the quick brown fox jumps.
2. MinHash signature. For each of m hash functions, record the minimum hash value across all shingles. Two documents with high Jaccard similarity will agree on many of these minima.
3. LSH banding. Split the m-dimensional signature into b bands of r rows each. Two documents become candidate duplicates if any band collides.
4. Verify. Compute exact Jaccard similarity only for candidate pairs.

The Jaccard similarity between two shingle sets A and B is:

MinHash has a beautiful property: the probability that two documents agree on any single row of their signatures equals their Jaccard similarity s. With b bands of r rows, the probability that two documents with similarity s become a candidate pair is:

For b=16 and r=4, a pair with s=0.8 is a candidate with probability:

A pair with s=0.3 is a candidate with probability:

That sigmoid-like curve is what makes LSH useful. Without LSH, comparing N documents pairwise costs O(N^2). With LSH, each document is hashed into a small number of buckets, and candidates are found by scanning buckets. The cost is near-linear in N.

Mixing sources with temperature

After filtering and deduplication, you have clean documents from several sources: web, books, code, academic papers, forums. You need a training mix.

The simplest approach is proportional sampling by source size: if 80% of your tokens are web and 20% are code, sample 80% web. But that treats every source as equal quality. In practice, code and books are higher quality per token than raw web, so labs deliberately upsample them.

A common recipe is temperature scaling. Let s_i be the available size of source i. The desired weight is:

• T=1 is proportional sampling.
• T < 1 sharpens the distribution toward the largest source.
• T > 1 flattens the distribution and upsamples smaller, often higher-quality sources.

For example, if web is 10x larger than code, T=0.5 makes the web weight \sqrt{10} \approx 3.2 times the code weight, while T=2.0 makes it 10^{0.5} \approx 3.2 as well? Wait: s_{\text{web}}^{1/2}=\sqrt{10}\approx3.16, s_{\text{code}}^{1/2}=1, ratio 3.16. For T=2, s_{\text{web}}^{1/2}? No, 1/T=0.5, same. Hmm. Actually for T=2, s_i^{1/T}=s_i^{0.5}, same as T=0.5 mathematically. That is a mistake in intuition; the exponent is 1/T, so T<1 gives exponent >1, sharpening; T>1 gives exponent <1, flattening. Let's redo: web=10, code=1.

• T=0.5: exponent=2, weights 100 and 1, ratio 100.
• T=1.0: weights 10 and 1, ratio 10.
• T=2.0: exponent=0.5, weights 3.16 and 1, ratio 3.16.

So lower temperature amplifies the dominant source; higher temperature brings rare sources closer to parity. Labs choose temperature by ablation, not by hand: train small models with several mixes and pick the one that minimizes perplexity on held-out evaluations.

Build It

Stage 1: Rule filter

The first stage is a configurable RuleFilter. Each rule returns a rejection reason, and the pipeline counts how many documents each rule removes. This produces the survival funnel.

pythonCopyfrom main import RuleFilter

config = {
    "min_words": 30,
    "max_symbol_ratio": 0.20,
    "max_caps_ratio": 0.20,
    "max_duplicate_line_ratio": 0.50,
    "max_duplicate_paragraph_ratio": 0.60,
    "min_stopword_ratio": 0.03,
    "ban_ellipsis_ending": True,
    "max_mean_word_length": 12,
    "max_lines_with_ellipsis_ratio": 0.20,
}

flt = RuleFilter(config)
reasons = flt.reasons("BUY NOW SALE DISCOUNT BUY NOW SALE DISCOUNT...")
print(reasons)  # ['max_caps_ratio', 'min_stopword_ratio']

The implementation in code/main.py is a direct translation of the Gopher rules into auditable Python. Every threshold is a named constant, so you can turn a rule off by removing it from the config or relax it when you are training on a domain like source code where symbol ratios are naturally higher.

Verify It

code/tests/test_main.py locks down the filter stage:

• every synthetic junk document is rejected by at least one rule;
• every synthetic clean document passes all rules;
• the per-rule drop counts plus the kept count are consistent with the input size.

Run them with:

consoleCopypython3 -m unittest discover tests -v

Build It

Stage 2: MinHash + LSH deduplication

The MinHashDeduplicator in code/main.py builds a 64-row MinHash signature and hashes it into 16 bands of 4 rows each. It then verifies candidate pairs with exact Jaccard similarity and returns duplicate clusters.

pythonCopyfrom main import MinHashDeduplicator

dedup = MinHashDeduplicator(
    num_hashes=64,
    bands=16,
    rows_per_band=4,
    shingle_size=5,
)

kept, indices, clusters = dedup.deduplicate(docs, threshold=0.8)

A cluster is a list of indices that are near-duplicates of each other. The deduplicator keeps the first document in each cluster and removes the rest.

Stage 3: Temperature mixer

The mixer takes a dictionary of source names to document lists and samples a target mix according to the temperature-scaled weights.

pythonCopyfrom main import temperature_mix

mix, counts = temperature_mix(
    {"web": web_docs, "code": code_docs, "discussion": discussion_docs},
    total=10_000,
    temperature=0.7,
)

A temperature of 0.7 is slightly sharper than proportional sampling: it favors the largest source while still preserving enough of the smaller sources for the model to learn their style.

The batch-of-strings API

The three stages are wrapped in CorpusCleaner, which accepts a list of strings and optional source tags and returns a CleaningResult.

pythonCopyfrom main import CorpusCleaner

cleaner = CorpusCleaner(mix_total=10_000, mix_temperature=0.7)
result = cleaner.clean(raw_docs, source_tags=tags)

This is the shipped artifact. The input can be the synthetic corpus built into main.py, or it can be a batch of Common Crawl WET documents read from disk. The API is the same.

Verify It

The end-to-end tests exercise the parts that make this a real pipeline:

• exact duplicates survive filtering and land in the same duplicate cluster;
• a high-Jaccard near-duplicate shares at least one LSH band bucket with its original;
• MinHash signature agreement approximates exact Jaccard similarity;
• lowering or raising the temperature changes the source counts of the final mix;
• the mixer never samples more documents than a source has available;
• deduplication keeps one representative per cluster.

Run the full demo with:

consoleCopypython3 main.py

You will see a survival funnel: how many documents pass the rule filter, how many duplicate clusters are removed, and how the final mix is split across sources.

Why This Matters for Engineers

• Corpus quality is the floor for small models. Architecture gets the headlines, but a 1B model trained on clean data beats a 3B model trained on raw crawl on most downstream tasks. Cleaning is the cheapest way to improve model quality.
• Cleaning is mostly deterministic engineering. You can write tests, version thresholds, and audit exactly why a document was removed. That is rare in a field dominated by stochastic training runs.
• Deduplication prevents memorization and regurgitation. Near-duplicate training data is the main cause of models quoting copyrighted text or repeating privacy policies verbatim. LSH makes that prevention cheap.
• Mixture design is an ablation problem, not a guess. Temperature scaling gives you a dial. The right setting is the one that wins on your evaluations, not the one that feels right.
• Any RAG corpus uses the same pipeline. Retrieval-Augmented Generation systems fail when the indexed documents are spam, duplicates, or boilerplate. The same filter, dedup, and mix logic applies at smaller scale.
• The survival funnel is your debugging tool. If a rule removes 40% of your data, inspect the rejects before you trust the rule. The counts tell you where the pipeline is too aggressive.

Hints

• If the rule filter removes most of your code data, raise max_symbol_ratio or add code-specific rules rather than disabling the filter entirely.
• A duplicate cluster of size two where both documents have very different lengths is often boilerplate wrapped around different bodies. Check the Jaccard threshold before removing.
• If temperature_mix returns the same counts for every temperature, your total is larger than the number of available documents. The mixer caps at source size, so it flattens all weights.
• When swapping in real Common Crawl, run the rule filter on a sample first and read the reject reasons. The thresholds in this lesson are tuned for English web text; other languages or domains need different defaults.

Key Terms