Data Leakage and Experiment Hygiene — When the Score Is Too Good, Suspect It

When a model scores better than you expected, that is not good news yet. This article splits data leakage into five concrete shapes — outcome-derived columns, preprocessing before the split, time running backwards, duplicates and groups, and a worn-out test set — then covers the detection moves that actually find them and the experiment records that let you dissect the day the number jumped.

The student whose mock exam score doubled

A student scores in the 99th percentile on a practice test. Last month they were at the median. Nobody can explain it — until someone notices that the practice test was the same worksheet handed out the week before, answer key included. The student did not get smarter. Only the measurement moved.

That is almost always what is happening when a machine learning model scores better than you expected. Data leakage is when information that would not actually be available at prediction time slips into training, validation, or model selection.

What makes leakage nasty is that it only ever pushes the score in the flattering direction. A normal bug makes numbers worse, and worse numbers get investigated. Leakage makes them better, so nobody investigates. It ships. The 0.95 AUC on your laptop becomes 0.6 in production, and because the fault lives in how you measured rather than in the model, no amount of model improvement fixes it.

Why we split data into train, validation, and test in the first place is covered in Overfitting and Evaluation Design. This article picks up one step later: the situation where you split it correctly and it is still broken. There are five shapes this takes.

One line that defines leakage

Before the taxonomy, keep exactly one test in your head.

Picture the instant this model serves a prediction in production. Using only the information you actually have at that instant, could you construct this feature?

If you could not, it is leakage. Which means hunting leakage is not a feat of cleverness — it is a timestamp audit. Walk the columns one at a time and ask when each value becomes known. That alone catches most of it.

Shape 1: Columns that grew out of the outcome

A churn model with a "visits to the cancellation page" column. A deterioration model with an "ICU transfer" flag. An equipment failure model with a repair ticket number. Every one of these is a column that only gets filled in after the event you are trying to predict has already happened.

In the database they sit on the same row as the label, perfectly innocent-looking. SELECT * picks them up. The model happily uses that column and nothing else, and returns a near-perfect score.

Finding them is unglamorous: list the columns and annotate each with does this value settle before or after prediction time? Watch especially for mutable tables that overwrite in place. If a customer table keeps only the current status and no history, then a row you pulled for last January is carrying today's value. In that case the entire column leaks for every row.

Shape 2: Preprocessing before the split

This is the trap that well-intentioned people step in. Take standardization.

Here $x_i$ is one feature's value on row $i$, $\mu_{\text{all}}$ and $\sigma_{\text{all}}$ are the mean and standard deviation computed over all the data, and $N$ is the total row count. In words: you normalized the whole dataset and then cut it into train and test.

Notice that test rows participated in computing $\mu_{\text{all}}$. The distribution of the test set has bled into training. The correct move is to compute $\mu_{\text{train}}, \sigma_{\text{train}}$ from training data only and apply them to the test set, learning nothing from it.

For standardization alone the damage is usually small. What makes it worth caring about is that the same mistake generalizes:

• Imputation — filling missing values with a median computed over everything
• Target encoding — the per-category mean of the target, computed over everything (this one touches $y$ directly, so it is severe)
• Dimensionality reduction — fitting PCA on everything
• Feature selection — keeping the top $k$ columns by correlation with $y$, measured over everything
• Oversampling — running SMOTE and then splitting

The last two do not merely inflate the score; they void the experiment. Generate 3,000 columns of pure noise, select the 20 most correlated with the target across the full dataset, then cross-validate: AUC can land north of 0.8. The features are random numbers. The selection step looked at $y$, and that $y$ included the validation rows, so the surviving 20 columns are simply "the noise that happened to resemble the answers." SMOTE fails the same way — oversample before splitting and synthetic points interpolated from test rows end up in training. That is copying the answer key with a blur filter on.