Building a Pretraining Corpus — From Web Sludge to Textbook Quality

Behind the single line "pretrained on a large corpus of web text" sit four stages: text extraction, quality filtering, deduplication, and mixture weights. This is a from-scratch walkthrough of how a gravel heap called Common Crawl gets sifted into textbook-quality prose — from the MinHash equation to the parameter names you actually touch.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

Start with panning for gold

A model card usually gives you one line: pretrained on a large corpus of web text. Hidden behind it are typically months of unglamorous sorting work.

Picture panning for gold in a river. You sink a bucket of gravel into the water, swirl it, let the light sand wash away, and keep only the heavy grains. A pretraining data pipeline does exactly this. The only difference is that the gravel heap is a few hundred terabytes.

This matters more than most architecture decisions. Same architecture, same compute budget — feed them different data and you get different models.

The raw material: Common Crawl as a gravel heap

Very few organizations can crawl the open web themselves. For most models the starting point is the web archive published by Common Crawl, a nonprofit. It releases a snapshot roughly once a month, each on the order of billions of pages.

It ships in three formats. WARC is the raw HTTP exchange, HTML included. WAT holds metadata only. WET is a naive text rendering with the tags stripped out.

WET is the easy option, and this is where the first fork in the road appears. WET extraction turns navigation menus, copyright notices, and ad copy into text without distinguishing any of it from the article. A page with three lines of real content can yield two hundred lines of menu. So the datasets that care about quality — Falcon's RefinedWeb, HuggingFace's FineWeb — throw WET away and redo extraction from the WARC HTML with a boilerplate-removal tool such as trafilatura.

The principle is worth stating plainly: the quality of your extraction sets the ceiling for every stage downstream. No filter later in the pipeline can recover an article that was never separated from its sidebar.

Get the orders of magnitude straight

How many tokens do you need? As covered in Scaling Laws from Scratch, the compute-optimal rule of thumb is roughly 20 tokens per parameter. Meanwhile RefinedWeb reports that after URL filtering, extraction, language identification, quality filtering, and deduplication, something like a tenth of the original survives. Plan on throwing away 90%.

The other wall is computational. Comparing a billion documents against each other pairwise is about 5×10¹⁷ comparisons. At a billion comparisons per second that is over fifteen years.

Once you accept that O(n²) is off the table, the direction is fixed: stop comparing everything, and narrow down to plausible candidates first.

Gate one: quality filtering

With body text extracted, the next question is whether a document is worth reading at all. The stages below get smarter as you go down the list — and more expensive.

(1) URL level. A domain blocklist drops adult sites, spam farms, and machine-generated content wholesale. It is the cheapest stage in the pipeline because you never read a single byte of the page.

(2) Language identification. A fastText language ID model scores each document and you keep the ones above a threshold. RefinedWeb's condition was an English score of 0.65 or higher. Raising the threshold raises purity, but technical documents with code mixed in, and genuinely multilingual pages, start falling out.

(3) Rule-based heuristics. The filter set published in DeepMind's Gopher paper has become the de facto standard. Documents get dropped for having an extreme word count, a mean word length outside roughly 3–10 characters, too high a ratio of symbols to words, a majority of lines beginning with a bullet character, or fewer than two of the basic English stop words (the, be, to, of, and, that, have, with).

Notice what these rules are not doing: they make no judgment about whether the content is any good. What they detect is the shape of a broken document. A keyword-stuffed SEO page, mojibake, a page that is nothing but a product listing — you can identify all of these without reading them.

(4) Statistical and learned filters. CCNet measured perplexity with a 5-gram language model (KenLM) trained on Wikipedia, then sorted documents into three buckets by how Wikipedia-like they looked. Push further and you reach classifiers. GPT-3 trained one using human-curated text as positive examples and raw Common Crawl as negatives. FineWeb-Edu had an LLM score pages for educational value, trained a small, cheap classifier on those scores, and used it to pull out 1.3T tokens. That progression is what "textbook quality" in the title actually refers to.

There is a real cost to this. Researchers have documented that blocklists of banned terms remove text from certain dialects and communities at disproportionate rates. However you design your filters, that design becomes the shape of the world your model gets to see.

Gate two: deduplication

The same text shows up on the web over and over. If a document appears n times in your corpus, you have effectively trained on that passage n times, whether you meant to or not. Lee et al. report that models trained on deduplicated data emit training text verbatim far less often. And if a benchmark's questions have been reposted somewhere in your crawl, your evaluation is contaminated and you will not find out until someone else runs your model.

Exact duplicates are easy at the whole-document level — just hash the document. The awkward case is documents that share only a portion of their text. The standard tool is a suffix array: find substrings shared across documents above some length (50 tokens, say) and excise them. One survey of C4 found the same 61-word English passage appearing over sixty thousand times.

Near-duplicates differ by a word, or by where the ad block landed, and exact hashing sails straight past them. This is where MinHash comes in. You convert a document into the set of its shingles — every run of n consecutive words, with n around 5 — and measure similarity by how much those sets overlap.

In words, the Jaccard coefficient is the fragments that appear in both documents divided by the fragments that appear in either one. Identical documents score 1, and documents with nothing in common score 0.

Here $A$ and $B$ are the shingle sets of two documents and $J$ is the Jaccard coefficient. In plain words: the fragments that appear in both, divided by the fragments that appear in either. Identical documents score 1; documents with nothing in common score 0.

The problem was never the formula — it was that you cannot evaluate it for every pair. So apply a hash function $h$ to every element of a set and keep only the minimum value. Then the following holds.

In words, the probability that the two minimum hash values happen to coincide is exactly the Jaccard coefficient. That lets you squeeze a variable-length document down to a short signature without comparing every pair directly.

In other words, the probability that the two minima happen to coincide is exactly the Jaccard coefficient. Every element of $A \cup B$ is equally likely to be the one that hashes lowest, and the minima match precisely when that element happens to live in $A \cap B$. That single fact lets you crush a variable-length document down to a short fixed-size signature.

import numpy as np

def minhash_signature(text, num_perm=128, ngram=5, seed=0):
    words = text.split()
    shingles = {" ".join(words[i:i+ngram]) for i in range(len(words)-ngram+1)}
    a, b = np.random.default_rng(seed).integers(1, 2**61-1, (2, num_perm))
    sig = np.full(num_perm, 2**61-1, dtype=np.int64)
    for s in shingles:
        x = hash(s) & 0x7FFFFFFF
        sig = np.minimum(sig, (a * x + b) % (2**61-1))   # min per hash function
    return sig

You also want to avoid comparing every pair of signatures, so you split each signature into $b$ bands of $r$ rows each and treat two documents as candidates only if some band matches exactly. This is LSH. The probability of becoming a candidate, $1-(1-J^{r})^{b}$, traces an S-curve against $J$, which means $b$ and $r$ are how you set "how similar counts as duplicate." The same machinery from a different angle is covered in Hashing and Nearest-Neighbor Search.

One piece of field wisdom to carry with you. The FineWeb team reported that deduplicating each snapshot independently produced better downstream results than deduplicating globally across all snapshots. Global dedup skews the survivors toward documents nobody ever bothered to repost, which is not the same thing as documents worth reading. More aggressive deduplication is not automatically better.

Gate three: how to mix

The last stage is the mixture weights. The Pile's contribution was largely a design stance — the web alone is not enough — expressed as a deliberate blend of 22 subsets: academic papers, GitHub, patents, subtitles, mathematics, and more.

And crucially, you do not weight sources in proportion to how much of them you have. In GPT-3's mixture table, filtered Common Crawl contributes 410B tokens of raw material but only 60% of the training mix, while Wikipedia has just 3B tokens and gets 3%. Work through the arithmetic and Common Crawl is seen for 0.44 epochs — never read to completion — while Wikipedia is read 3.4 times. The small, high-quality pile is deliberately read more heavily.

For multilingual data, this allocation is usually set by a formula.

In words, each language's sampling probability is its data amount raised to the power α, divided by the same quantity summed over all languages. Lowering α boosts low-resource languages, while α = 1 samples in proportion to raw volume and α = 0 treats every language equally.

$n_i$ is how much data you have for language $i$, $p_i$ is the probability of sampling from it, and $\alpha$ is an exponent between 0 and 1. At $\alpha=1$ you sample in proportion to raw volume; at $\alpha=0$ every language is treated equally (mT5 and XLM-R settled around 0.3). The short version: the lower you push $\alpha$, the more low-resource languages get boosted. That "sharpen it or flatten it" intuition is precisely the temperature knob on a softmax.

More recent work also reports raising the proportion of high-quality data during the late phase of training, a technique usually called annealing. It is the same instinct as drilling nothing but past exam papers the week before the test.

Two steps before shipping, and the epoch ceiling

Two stages are non-negotiable before a corpus goes out the door. Decontamination removes training documents that overlap an evaluation set by more than a fixed n-gram length — GPT-3 used 13-grams. PII removal masks email addresses, phone numbers, and similar identifiers by regular expression.

Then there is the question everyone eventually hits: when you run out of data, how many times can you go around again? Muennighoff et al.'s scaling laws under data constraints report that repeating data for up to roughly four epochs is nearly as valuable as adding fresh data, and past that the value falls off sharply. "Just train more epochs" holds — but only within limits, and you should know where they are.

How this is used in practice

Who does this work. Data engineers on teams building foundation models, and consultancies running continued pretraining for a specific domain — medical, legal, internal company documents. The second case is smaller in scale but structurally identical, and internal document collections are so full of duplicates that deduplication pays off unusually well there.

Tools you will touch. trafilatura for boilerplate removal, fastText's lid.176 for language ID, KenLM for perplexity, datasketch or text-dedup for MinHash/LSH, and HuggingFace's datatrove or AI2's dolma for the pipeline as a whole.

Parameter names you will tune. num_perm (number of hash functions, around 128), threshold (Jaccard cutoff, typically 0.7–0.85), ngram_size (shingle length, around 5), min_doc_words / max_doc_words, the language-ID score threshold, and the mixture alpha.

Pitfalls that turn into incidents:

• Settling for WET. Extraction quality is the ceiling on everything downstream, and no later stage can raise it.
• Getting the stage order wrong. Run expensive deduplication before the cheap filters and you spend compute on documents you were going to discard anyway.
• Not recording the seed. Change the hash seed and a different set of documents survives. A dataset is only an asset once it reproduces.
• Skipping decontamination. Your internal benchmark scores jump implausibly, and then collapse the moment anyone evaluates externally.
• Swapping the tokenizer late. Every token-count estimate you made silently becomes wrong. Size your data after the tokenizer is frozen, not before.
• Never reading the rejects. Filter bias is invisible in aggregate metrics. The only way to find it is for a human to sit down and read a sample of what got thrown away.

How this shows up as a design question. "Design deduplication for a trillion-token corpus." The path through it: shingle the documents, build MinHash signatures, band them with LSH to produce candidate pairs, union-find the candidates into connected components, and keep one document per cluster. To distribute it, use the band hash values as the shuffle key.

Wrapping up

• The pipeline is four stages — extraction → quality filtering → deduplication → mixture weights. The further upstream, the cheaper the stage and the larger its effect.
• Start quality filtering with rules that detect the shape of a broken document, then escalate to statistics and classifiers.
• Deduplicate by narrowing to candidates with MinHash and LSH — but remember that more aggressive is not automatically better.
• Do not set mixture weights in proportion to volume. Read the small, high-quality piles more heavily.

At this point you have a model that can continue a piece of text, but it will not follow instructions yet. That step is covered in Instruction Tuning and RLHF.