What Ratio Should the Validation Split Be — I Swept It, and What Mattered Was the Count, Not the Ratio

When you split your data into training and validation, is it 80/20 or 70/30? I ran model selection over 30 candidates 2,000 times per condition and swept the validation share from 5% to 50%. There is a valley — but changing the ratio tenfold moves the loss by only 1.2x, while holding the ratio fixed and changing the count fourfold moves it 6.5x. Refitting on all the data with the chosen setting erases the valley entirely. When you cannot spare a few hundred rows, k-fold wins on average, but it is a coin flip on any single run.

What exactly are the 8 and the 2 in "80/20" deciding?

You have 1,000 labelled rows. You want to try a handful of model settings — number of layers, polynomial degree, strength of the penalty — and keep the best one. How many rows do you set aside to do the keeping? 80/20, 70/30, 90/10. All of them are common.

So what is that ratio improving? Whatever goes to validation cannot be used for training. But if validation is too small, the selection itself becomes unreliable. You are squeezed from both sides — that much is clear. What is not clear is where the bottom of the valley sits, and how much it costs you to be off it.

This article measures exactly that one thing. The answer up front: the valley is real, but the ratio was the wrong unit.

How it was measured

Fix a true function and draw noisy observations from it.

Here $x$ is drawn uniformly from $-1$ to $1$ and the noise is $\varepsilon \sim N(0, 0.30^2)$.

In words: every observation is a smooth curve — a sine wave riding on a gentle upward slope — with random jitter of typical size 0.30 laid on top. The curve is what a model can learn; the jitter is what no model can ever predict.

There are 30 candidates: polynomial degree $d = 1,\dots,10$ crossed with a penalty $\lambda \in \{10^{-6}, 10^{-4}, 10^{-2}\}$, each fitted by ridge regression. The true function is not a polynomial, so neither "higher degree is better" nor "lower is better" holds. There is a sweet spot somewhere, and the validation data has to find it — that search is what this article measures.

Quality is reported as generalisation error. Drawing a random test set would mix in the test set's own jitter, so instead the error over all of $x$ is integrated numerically (400-node Gauss–Legendre quadrature) and the noise term $\sigma^2$ added. As a cross-check, the same coefficients measured on a random test set of 100,000 rows give 0.3045 against 0.3046 from the quadrature.

With this setup, nothing can ever go below 0.3000: the noise itself is the floor. The tables below also show the excess over that floor (RMSE − 0.3000), which is where the actual skill difference lives.

Every condition redraws the data 2,000 times and averages. The random seed is fixed.

Sweeping the share from 5% to 50%

Total $N=200$ rows. Sweep the validation share $v$. Here the chosen model is used as is — trained on the training part only and shipped.

The valley is at 25–30%. The standard error of the mean runs 0.0001 to 0.0005, so these differences are large enough to read.

Two things to take from it.

Both ends are bad. At 5% the excess is 0.0152, which is 1.79x the valley. At 50% it is 0.0101, or 1.19x. The reasons differ by side. On the left, validation has only 10 rows, so the selection misfires. On the right, training is down to 100 rows, so the model itself is weaker. Same curve, two different accidents.

The middle is flat. Between 15% and 40% the excess is 0.0085 to 0.0097 — at most 14% away from the valley. The shape repeats at $N=60$ and $N=1{,}000$.

Squeezing validation to 5% costs 1.5–1.8x in excess error across three orders of magnitude of $N$, and the valley stays near 25% in all three.

Misfiring is normal. The size of the loss is the question

Count the misfires directly: how often the pick equals the candidate with the genuinely lowest generalisation error among the 30 (knowable only after the fact).

At $N=200$ with 10 validation rows, the pick is right 10.1% of the time. With 100 validation rows it is 30.4%. So roughly seven runs in ten miss the true best.

The excess still stays around 0.01 because missing is cheap. Measure the gap between the chosen candidate and the true best (call it the selection loss).

Many of the 30 candidates are effectively the same model. Counting them, 5.1 on average sit within 0.001 of the best and 13.8 within 0.005 ($N=200$, 160 training rows, 1,000 repetitions). Degree 7 with $\lambda = 10^{-6}$ versus $10^{-4}$ is nearly the same function. So a "miss" is not itself an accident. Watch the tail. With 10 validation rows, one run in a hundred loses 0.0673, which multiplies the excess over the floor by about five. With 100 rows that 99th percentile shrinks to 0.0176. The tail heals before the mean does when you add rows.

Refit on everything, and the valley disappears

So far the model trained on the training part was shipped as is. Switch to the other common practice: take the chosen setting and retrain on training plus validation combined. Same experiment, same conditions.

The valley is gone, and larger shares are simply better, monotonically out to 50% (and flat past 30%: 0.3075 to 0.3072). At $N=60$ and $N=1{,}000$ the refit minimum also lands at 50%.

The reason only needs saying once. Refitting returns the training rows you lent out, so the ratio now controls only the quality of the selection — and that improves with more validation rows. The valley existed because of one side of the trade-off alone, and refitting removes that side.

In other words, "80/20 or 70/30" has no answer until you have decided whether you will retrain with the chosen setting.

Hold the ratio fixed, move the total, and nothing happens

This is the point. If the ratio is a suspect unit, hold the validation count fixed and move the total $N$. The quantity measured is the selection loss.

In the top four rows the share changes tenfold, from 12.5% to 1.2%, and the loss moves from 0.0053 to 0.0044 — a factor of 1.2.

Now read it the other way. Two rows have the same 5% share: 25 validation rows at $N=500$ loses 0.0052, and 100 validation rows at $N=2{,}000$ loses 0.0008. Same share, 6.5x the loss.

Holding the total fixed and moving only the count gives the same shape ($N=1{,}000$).

The count decides the quality of the selection. The ratio is what falls out after you have chosen the count. Ask "what ratio?" and you hand the same answer to two projects whose $N$ differs tenfold — which, per the table above, is off by 6.5x.

When you cannot spare a few hundred rows — k-fold

Deciding by count is fine until the total is small. At $N=60$ you cannot set aside 200 rows. This is where k-fold comes in: split into $k$ parts, train on $k-1$ and score all candidates on the held-out one, rotate $k$ times and sum, then retrain the winning setting on everything. Validation rows are reused rather than lent away, so all rows serve both selection and training.

Compare against a fixed split (80/20, chosen setting refit on all). The comparison is paired — both methods run on the same draw, which removes luck of the draw from the difference. 3,000 repetitions.

At $N=60$ the excess over the 0.3000 floor goes from 0.0336 to 0.0235 — down 30%. At $N=200$ it is 0.0087 to 0.0065, and at $N=1{,}000$ it is 0.0016 to 0.0011, which is negligible. k-fold earns its keep only while the data is small, and the gain fades as $N$ grows. $k=10$ was measured in the same run (0.3242 / 0.3065 / 0.3011); paired on the same draws, its gap from $k=5$ is 0.0003 at $N=60$ (standard error 0.0003) and 0.0000 at $N=200$ and $N=1{,}000$ — invisible.

How it wins matters, though. k-fold wins the individual run only 57.1% / 51.9% / 58.5% of the time. It is clearly ahead on average yet near even head to head, because the difference is made by the occasional run where the fixed split misses badly. Try to confirm "k-fold helped" from a single experiment and half the time you will conclude the opposite.

The cost is compute: $k$-fold fits every candidate $k$ times, so the naive implementation takes $k$ times as long.

How to decide

Only what the measurements support.

1. First decide whether you will refit on everything with the chosen setting. If you will, a large share (30–50%) costs you nothing. If you will ship the model as is, the valley is near 25%.
2. Then decide the validation count, not the ratio. Here, 100 rows brought the selection loss to 0.0011 and the rate of losses above 0.0100 down to 0.2%. The amount you need depends on how many candidates you have and how close they are, so rebuilding this table with your own candidate set is the fastest way to know.
3. If that count still leaves plenty for training, stop worrying about the ratio. Anything from 15% to 40% was within 14% of the valley.
4. If you cannot spare it, use k-fold. At $N=60$ the excess dropped 30%. It costs $k$ times the compute, and a single experiment cannot verify it.

The core, for reproduction:

import numpy as np

# 30 candidates: degree 1..10 x 3 penalties
CANDS = [(d, l) for d in range(1, 11)
         for l in (1e-6, 1e-4, 1e-2)]

# ridge regression, polynomial basis
def fit(x, y, d, lam):
    X = np.vander(x, d + 1, increasing=True)
    A = X.T @ X + lam * np.eye(d + 1) * len(x)
    return np.linalg.solve(A, X.T @ y)

# hold out the last nv rows and pick one
def pick(x, y, nv):
    xtr, ytr = x[:-nv], y[:-nv]
    xv, yv = x[-nv:], y[-nv:]
    err = []
    for d, l in CANDS:
        w = fit(xtr, ytr, d, l)
        V = np.vander(xv, d + 1, increasing=True)
        r = V @ w - yv
        err.append(np.sqrt(np.mean(r * r)))
    # then retrain this setting on everything
    return CANDS[int(np.argmin(err))]

How it is used in practice

Who decides this, and when. A data scientist or ML engineer building a regression or classification model on tabular data, at the start of a project, while setting up the evaluation harness. The split chosen here becomes the yardstick for every comparison that follows; change it midway and the earlier numbers stop being comparable.

What you actually touch.

• In scikit-learn, train_test_split(test_size=...) treats a float as a share and an integer as a row count. Given the result above, passing a count keeps your intent intact.
• GridSearchCV and RandomizedSearchCV default to refit=True, meaning they retrain the winning setting on all the data you passed in. You are already on the "refit on all" side of the table above.
• KFold(n_splits=5) defaults to shuffle=False. If the rows are still sorted by date or by label, the folds will be lopsided. Switch to StratifiedKFold to balance class ratios, GroupKFold to keep one person or one store on one side, and TimeSeriesSplit to cut on time.
• In Keras, model.fit(validation_split=0.2) takes the last 20% of the rows before shuffling. Pass the data in its original order and validation becomes "only the latest period" or "only the last class".

Traps that cause real accidents.

• Fitting preprocessing on everything before the split. Compute scaling or imputation statistics on all rows and then split, and validation information leaks into training, making the selection look better than it is. Put preprocessing in a Pipeline so it is fitted inside each split.
• Reporting best_score_ as the model's ability. That number was spent on choosing (how much optimism rides on it is the separate problem noted below). Keep a set that the selection never touched for the number you report.
• Settling k-fold versus a fixed split with one experiment. As the table above showed, a single head-to-head is close to a coin flip. Pair both methods on the same draws and repeat.

Questions asked in interviews and design reviews, and how to reason through them.

• "Will you split 80/20 or 70/30?" → First ask whether the chosen setting will be retrained on everything. Then think in validation rows, not a ratio, and check that training still has enough after you take them. If it does not, use k-fold.
• "What k would you use for k-fold?" → It buys accuracy at $k$ times the compute, and pays most when the data is small. In this article's range the gap between $k=5$ and $k=10$ was invisible, and with large data even the gap to a fixed split nearly vanishes.
• "Why does the cross-validation score disagree with production?" → Suspect, in order: optimism from a number used for selection, a mismatch between how you split (time, groups) and how production arrives, and leakage through preprocessing.

What this article does not answer

Three things.

Whether the validation number itself can be reported as the model's ability. That is a separate problem. This article measured how good the chosen model is using a yardstick never involved in the selection (the numerically integrated generalisation error). No such yardstick exists in practice. What you actually hold is the validation number, and that number was spent on choosing. How much optimism rides on it is measured by Chapter 2 of Part III of the textbook, with an experiment of the same shape: selecting among 90 candidates on 10 validation rows made the winner's validation number look 0.34 better than its real ability. Chapter 2, "Fitting the Sample Too Well — Overfitting" is the continuation. Read it and you will also see that the tables above are only half of the answer to "why three splits?".

Stratification and time order. Here $x$ is drawn independently, so the content of the split — balancing class ratios, cutting on time, keeping one person or one store on one side — has no effect at all. On real data that is what bites first. What leaks in when the split is wrong is sorted into five shapes in Data Leakage and Experiment Hygiene.

The number of candidates. It was fixed at 30. More candidates make selection harder and raise the validation count you need. The figures here carry over only while your candidate set is in the tens. Searching over hundreds or thousands of candidates is covered in Hyperparameter Search.