Building a Pretraining Corpus — From Web Sludge to Textbook Quality

Behind the single line "pretrained on a large corpus of web text" sit four stages: text extraction, quality filtering, deduplication, and mixture weights. This is a from-scratch walkthrough of how a gravel heap called Common Crawl gets sifted into textbook-quality prose — from the MinHash equation to the parameter names you actually touch.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

Start with panning for gold

A model card usually gives you one line: pretrained on a large corpus of web text. Hidden behind it are typically months of unglamorous sorting work.

Picture panning for gold in a river. You sink a bucket of gravel into the water, swirl it, let the light sand wash away, and keep only the heavy grains. A pretraining data pipeline does exactly this. The only difference is that the gravel heap is a few hundred terabytes.

This matters more than most architecture decisions. Same architecture, same compute budget — feed them different data and you get different models.

The raw material: Common Crawl as a gravel heap

Very few organizations can crawl the open web themselves. For most models the starting point is the web archive published by Common Crawl, a nonprofit. It releases a snapshot roughly once a month, each on the order of billions of pages.

It ships in three formats. WARC is the raw HTTP exchange, HTML included. WAT holds metadata only. WET is a naive text rendering with the tags stripped out.

WET is the easy option, and this is where the first fork in the road appears. WET extraction turns navigation menus, copyright notices, and ad copy into text without distinguishing any of it from the article. A page with three lines of real content can yield two hundred lines of menu. So the datasets that care about quality — Falcon's RefinedWeb, HuggingFace's FineWeb — throw WET away and redo extraction from the WARC HTML with a boilerplate-removal tool such as trafilatura.

The principle is worth stating plainly: the quality of your extraction sets the ceiling for every stage downstream. No filter later in the pipeline can recover an article that was never separated from its sidebar.

Get the orders of magnitude straight

How many tokens do you need? As covered in Scaling Laws from Scratch, the compute-optimal rule of thumb is roughly 20 tokens per parameter. Meanwhile RefinedWeb reports that after URL filtering, extraction, language identification, quality filtering, and deduplication, something like a tenth of the original survives. Plan on throwing away 90%.

The other wall is computational. Comparing a billion documents against each other pairwise is about 5×10¹⁷ comparisons. At a billion comparisons per second that is over fifteen years.

Once you accept that O(n²) is off the table, the direction is fixed: stop comparing everything, and narrow down to plausible candidates first.

Gate one: quality filtering

With body text extracted, the next question is whether a document is worth reading at all. The stages below get smarter as you go down the list — and more expensive.

(1) URL level. A domain blocklist drops adult sites, spam farms, and machine-generated content wholesale. It is the cheapest stage in the pipeline because you never read a single byte of the page.

(2) Language identification. A fastText language ID model scores each document and you keep the ones above a threshold. RefinedWeb's condition was an English score of 0.65 or higher. Raising the threshold raises purity, but technical documents with code mixed in, and genuinely multilingual pages, start falling out.

(3) Rule-based heuristics. The filter set published in DeepMind's Gopher paper has become the de facto standard. Documents get dropped for having an extreme word count, a mean word length outside roughly 3–10 characters, too high a ratio of symbols to words, a majority of lines beginning with a bullet character, or fewer than two of the basic English stop words (the, be, to, of, and, that, have, with).

Notice what these rules are not doing: they make no judgment about whether the content is any good. What they detect is the shape of a broken document. A keyword-stuffed SEO page, mojibake, a page that is nothing but a product listing — you can identify all of these without reading them.

(4) Statistical and learned filters. CCNet measured perplexity with a 5-gram language model (KenLM) trained on Wikipedia, then sorted documents into three buckets by how Wikipedia-like they looked. Push further and you reach classifiers. GPT-3 trained one using human-curated text as positive examples and raw Common Crawl as negatives. FineWeb-Edu had an LLM score pages for educational value, trained a small, cheap classifier on those scores, and used it to pull out 1.3T tokens. That progression is what "textbook quality" in the title actually refers to.