Chapter 1 chose to fit the rule rather than write it. More fitting power looks like a better deal, and up to a point it is: push it harder and performance on the data in your hands keeps improving. Past that point, it improves only on the data in your hands.

That gap is this chapter. The cause is not the model's quality. It is that the quantity you are lowering and the quantity you want lowered are different things.

(Chapter 1 promised "Chapter 2" would cover why that loss — why cross-entropy. That needed the language of probability, so it moved to Chapter 0-3. This chapter takes the other half: what happens when you fit harder.)

1. What you lower vs. what you want lowered

What you actually want low is the error on data you have not seen. With pairs drawn from a distribution $\mathcal{D}$,

the generalisation error. Equation (1) is an expectation (Chapter 0-3, equation (3)), and since $\mathcal{D}$ is unknown you cannot evaluate it.

What you can evaluate is the average over your $n$ examples:

For a fixed $f$, equation (2) is an unbiased estimate of equation (1) (Chapter 0-3, equation (5)). The trouble is that training does not hold $f$ fixed. The moment you choose $f$ to make $\hat{R}$ small, $\hat{R}$ stops being a fair estimate of $R$. You are measuring a choice with the same ruler you chose by — that is what overfitting is, and the same story runs all the way to this chapter's common misreading.

2. Measured — what raising the degree does

Draw 20 points from $y = \sin(2\pi x) + \varepsilon$ with noise of standard deviation $\sigma = 0.25$, fit a degree-$d$ polynomial by least squares, and test on 4,000 fresh points.

Training RMSE never once rises. More capacity can always fit what is in front of it better. Test error, meanwhile, bottoms out at $d=5$ and then takes off. At $d=19$ the training error is exactly zero — 20 points threaded by 20 parameters — while the test error has six digits.

Test error cannot go below $\sigma = 0.25$. The $y$ values carry noise, so even naming the true $\sin(2\pi x)$ exactly leaves 0.25 behind. The 0.2947 at $d=5$ is sitting near that floor (the same role $H(p)$ played in Chapter 0-4).

The figure's numbers will not match the tables. It draws its own sample, so the same $d$ and $n$ give different values — $d=9,\ n=20$ reads 0.4457 in the table above and 47.8 in the figure. What matches is the shape: training falls forever, test is U-shaped, more data closes the gap. The mismatch is itself a result. Where the parameter count approaches the sample size, changing the sample changes the order of magnitude of the test error. It is a signal not to compare unstable things on a single measurement.

3. The size side — there is no such thing as "a model that overfits"

Hold $d = 9$ fixed and change only how many training points there are.

The model did not change at all. It still breaks at $n = 10$ and converges to within 0.005 at $n = 1000$, both curves near the 0.25 floor.

The $n = 10$ row is extreme because there are 10 parameters for 10 points. Whenever $n \le d+1$ there exists a solution threading every point: zero error where the points are, and no constraint whatsoever between them, so it runs wild there.

4. Reading the symptoms

Put the two errors side by side and the next move is determined (measured at $n = 20$).

In the $d=1$ row the gap is $-0.0278$: the test error is lower than the training error (0.5165 against 0.5443). The sign flipping is sample noise — another draw puts it near $+0.06$. The point is not the sign but that underfitting leaves the gap small while both numbers stay high. What to read is not the gap but how high both are, compared against the floor $\sigma$. Watch only the gap and you will call an underfitting model good "because it isn't overfitting."

5. Remedies

Roughly in decreasing order of reliability.

1. Add data. As the table in section 3 shows, this is the surest one. It fails only against underfitting.
2. Lower the capacity. Fewer degrees, layers, width. Effective — and overshooting lands you in underfitting.
3. Regularise. Penalise the size of the weights and squeeze capacity continuously.

Keeping $d = 15$ (the setting that broke in section 2) and moving only $\lambda$:

$67.45 \to 0.3071$ — back to roughly the $d=5$ result (0.2947) without lowering the degree. In the rows where $\lambda$ is too large (1, 10) even the training error rises: that is the picture of having gone too far into underfitting.

Equation (3) also reads in the language of probability. Take the maximum likelihood of Chapter 0-3, add a prior $w \sim \mathcal{N}(0, \tau^2 I)$ on the weights, and maximise the posterior (MAP):

L2 regularisation just is the prior "the weights should be near zero." The correspondence is $\lambda = \sigma^2/\tau^2$ with $\sigma$ the noise level: the more strongly you believe it (smaller $\tau$), the larger $\lambda$.

4. Early stopping. Stop part-way through training, because the number of steps is a form of capacity too. Where to stop is decided on validation data — and the hazard in that is the next section.
5. Data augmentation, dropout and the rest come in Chapter 4 onward.

A common misreading: "I picked it on validation data, so that number is the real performance"

Splitting data into train, validation and test is well-known practice. But why is two not enough? What was said under equation (2) applies unchanged: the ruler you chose with is not a fair ruler for the thing you chose.

Measured. Fit 90 candidates (15 degrees × 6 values of $\lambda$) on 30 training points, pick the best on validation, then measure that model on 4,000 test points. Averaged over 300 repetitions:

With 10 validation points, the winner's number looks 0.34 better than the truth. More validation data and the gap disappears.

That this is not coincidence is settled by removing the choosing. With the same 20 validation points but a single fixed candidate ($d=5$, $\lambda=10^{-2}$), the mean gap is $+0.0049$ — essentially zero, against $+0.1002$ under the same conditions with selection. The bias comes from the act of choosing, not from the split.

It grows with the number of candidates (20 validation points):

The more you try, the less the validation number means. The validation set has been spent on choosing, so reporting performance needs one more split that has never been used to choose anything. That is why there are three.

Exercises

The answers are folded away. Work them first.

Exercise 1. Twelve training points, fitting a degree-11 polynomial. (a) How many parameters? (b) What happens to the training error? (c) Is that good news?

Exercise 2. Two models measured on $n$ points. A: train 0.52, test 0.54. B: train 0.10, test 0.45. The noise is $\sigma = 0.25$. (a) The symptom in each. (b) The remedy for each. (c) Is "the smaller the gap, the better the model" right?

Exercise 3. You tried 90 candidates and chose on 20 validation points. The winner's validation RMSE is 0.25. (a) May you report 0.25 as its performance? (b) Roughly where is the real figure? (c) Which helps more: more validation data, or fewer candidates?

What this chapter does not say

• Double descent is not covered. The tables stop inside the range where raising $d$ keeps raising test error. The reported phenomenon of error falling again far past the point where parameters exceed examples cannot be shown in this setting (unregularised polynomials).
• Theoretical generalisation guarantees (VC dimension, PAC learning, uniform convergence) are untouched. This chapter is measurements only.
• Cross-validation — reusing data through $k$ splits — is not described. It is the standard remedy when validation data is scarce, and it acts on the top rows of the misreading table.
• The regularisation that actually does the work in deep learning (batch normalisation, augmentation, dropout, and the implicit bias of the optimiser itself) is Chapter 4 onward. The L2 here was chosen to show the mechanism in its smallest form.
• Every number was computed and checked (the three tables over degree, size and $\lambda$; the symptom table; the selection optimism over 300 repetitions together with the no-selection control; and the candidate-count sweep). The estimator is the same throughout: least squares in a basis with $x$ mapped to $[-1,1]$, 4,000 test points, one seed.

Going deeper

• Overfitting and evaluation design — be suspicious of 99% accuracy
• Weight initialisation and regularisation — what makes training start, and keep going
• Statistical learning theory — why learning generalises at all
• Loss functions and optimisation — how a model learns from being wrong