Overfitting and Evaluation Design — Be Suspicious of 99% Accuracy

Why a model that fits its training data perfectly falls apart in production: the division of labour between train, validation and test, cross-validation, the data leakage that ruins more projects than any modelling mistake, and how accuracy lies on imbalanced data.

No report is more dangerous than "we hit 99% accuracy"

Someone on the team reports that the new model is 99% accurate. Before celebrating, there are three things to check. Which data was it measured on? What is the class breakdown of that data? And was any of it used for training before it was used for measuring?

Get those three wrong and the model that scored 99% on your laptop will score 60% in production. The nasty part is that the cause lives in how you evaluated, not in the model — so no amount of model improvement will fix it.

The previous article (Loss Functions and Optimization) showed the machinery for driving the loss down. But driving the loss down is not the goal in itself, because training loss can be pushed down far enough to do harm.

What is actually happening when a model overfits

Overfitting is the state where a model has faithfully reproduced not only the real pattern in the data, but also the noise that happened to be sitting on top of it.

Picture a scatter plot with ten points: a gently rising trend, each point nudged up or down by measurement wobble. Fit a straight line and there is residual error, but the trend is captured. Fit a ninth-degree polynomial and the curve passes exactly through all ten points — training error zero. Yet between the points that curve swings wildly, and on a new point it misses badly.

So overfitting is not "poor performance". It is being too good on the training data. It is the symptom you get when a model's expressive capacity is large relative to how much data you have.

Drag the degree slider below. Training error falls monotonically as the model gets more complex, while test error turns around at some point and starts climbing.

The opposite failure exists too. Force a model that can only draw straight lines to learn a complicated relationship and both errors stay high; that is underfitting. Two numbers are enough to diagnose which one you have: both high means underfitting; training low with a widening gap (the generalization gap) means overfitting.

These two failure modes have names. The part of the error that comes from being too simple to capture the trend is bias; the part that comes from chasing the noise is variance. More complexity lowers bias and raises variance, and the minimum of the test error curve above is the bottom of that trade-off. The standard working order is: first use a model with enough capacity to drive training error down, then close the gap with regularization and more data.

Notice that the best complexity is not where training error is lowest but where test error is lowest. Which leaves exactly one question: how do you measure that test error honestly? That is what the rest of this article is about.

Train, validation, test — why three, not two

You split the data three ways, because the three have genuinely different jobs.

Why separate validation from test? Hyperparameters — learning rate, number of layers, regularization strength; the settings training does not determine — are chosen by looking at validation scores. But the act of choosing is itself a fit to the validation set. Try fifty configurations and pick the best, and that winner may simply have matched the validation set's noise. So you seal away data that was never used for any choice. The test set is an envelope you open once, for the report.