Building a Dataset in Practice — Collect, Clean, Blend

Most of the work in building a model is really the work of building its dataset. From designing the population you sample from, to calibrating filter thresholds, measuring leakage into your test set, converting mixture ratios into epochs, using synthetic data where it belongs, and writing annotation guidelines that actually hold — explained from zero, with the formulas and the procedures.

Self-Instruct: Aligning Language Models with Self-Generated Instructions

The prep work decides the dish

Same recipe, same heat, same pan — and yet the dish changes with the cook. What changes is the prep: where the ingredients came from, how the bad parts were trimmed, what got combined in what proportion. Most of the outcome is settled before anything hits the flame.

Model development has the same shape. Changing what a model eats often moves results further than adding another layer. Yet the conversation in most teams stops at "let's get more data." Getting more is one of three jobs, and it is the least effective one.

This article is about designing a dataset for a task you actually have. The web-scale story — turning a crawl into a pretraining corpus — is covered separately in How pretraining data is made. Whether you need tens of thousands of examples for a classifier or a few thousand for instruction tuning, the structure of the decisions is identical.

The shape of the work — four questions

Building a dataset means answering four questions, in order.

1. What do we collect? — designing the population
2. What do we throw away? — quality filtering and deduplication
3. How do we blend it? — mixture ratios
4. Who labels it? — annotation design

One principle governs the ordering: put the cheap, effective decisions first. Any heavy processing you spend on a document you were going to discard later is compute thrown away. Drop what you can drop by domain without reading a byte; then the counting checks (length, character ratios); then classifiers and humans last. That reordering alone changes cost by an order of magnitude.

Collect — not "a lot," but "from which distribution"

Before you collect anything, decide the population, not the volume. How closely can you match the distribution of inputs your system will actually see in production? If that is off, no amount of downstream cleaning will fix it.

Collect opportunistically and you will always skew toward whatever is easy to get: English, public data, weekday-daytime tickets, long documents. So fix your axes first — source, language, difficulty, document length, time period. Build a table from those axes and assign a minimum count to each cell. That is stratification. Any cell you leave empty becomes a region the model has never seen.

The other thing that pays off later is provenance. For every item, keep the source, the fetch date, the license, and the original URL. "Please delete everything that came from this provider" is a routine request even at small scale. If you blend first and cannot trace items back, you rebuild the dataset from scratch.

The intake checkup

Before cleaning, look at what you actually have: item counts, the length distribution, counts per source, the language breakdown, the rate of mojibake. If something looks wrong here, fixing the collection step is usually faster than writing a filter to compensate.

A useful move at this stage is to embed the documents and browse their neighborhoods. Copy-paste clusters, pages mass-produced from one template, and near-identical reposts all show up as tight neighbor groups. It is also worth feeling how the distance metric — dot product, cosine, Euclidean — changes who your neighbors are.

Clean, part 1 — quality filtering is threshold calibration

Quality filters get smarter in stages. Start with rules (length, symbol ratio, language ID), then statistics, then a classifier — these days most often a lightweight classifier trained on scores produced by an LLM acting as a judge.

But at every stage the mechanism is the same: assign each document a score $s(x)$ and draw a line at a threshold $\tau$. So the real work is not writing the filter. It is choosing $\tau$.

Here $s(x)$ is the score of document $x$, $y$ is a human judgment of quality, and $\pi(\tau)$ is the purity of what survives the threshold. In words: of everything you keep at threshold $\tau$, what fraction is actually good? Raise $\tau$ and purity goes up while volume goes down.

The procedure is unglamorous. Sample 200–500 items at random, label them by hand, sort by score, and build a table of purity and retention for each candidate $\tau$. Then decide between "88% pure, 60% retained" and "92% pure, 25% retained" with the numbers in front of you. Do not pick 0.8 because it feels right. Without that table, when results come out badly you cannot tell whether the threshold or the filter was at fault.

Build in one more step: a human reads the rejected side. Dialects, jargon-dense documents, prose with code mixed in, answers that are short but correct — whether those are being dropped unfairly is invisible from the surviving side. Filter bias becomes, directly, the bias in the world your model sees.

Clean, part 2 — duplicates, and leakage into the test set

Leaving duplicates in causes three separate harms: memorization, because the same document gets trained on repeatedly; inflated scores, because the same content sits on both sides of your split; and plain wasted compute.