#overfitting

3 articles

01 ·Machine Learning Basics·FREE·12 min read What Ratio Should the Validation Split Be — I Swept It, and What Mattered Was the Count, Not the Ratio When you split your data into training and validation, is it 80/20 or 70/30? I ran model selection over 30 candidates 2,000 times per condition and swept the validation share from 5% to 50%. There is a valley — but changing the ratio tenfold moves the loss by only 1.2x, while holding the ratio fixed and changing the count fourfold moves it 6.5x. Refitting on all the data with the chosen setting erases the valley entirely. When you cannot spare a few hundred rows, k-fold wins on average, but it is a coin flip on any single run. 02 ·Probability & Statistics·★ MEMBER·PAPER·11 min read Statistical Learning Theory — Why Does Learning Generalize? You only ever see finitely many examples, yet the model answers questions about data it has never seen. This piece builds the guarantee from scratch — Hoeffding, uniform convergence, VC dimension, PAC — then shows where it broke against deep learning, and what survives in day-to-day evaluation design. 03 ·Machine Learning Basics·★ MEMBER·8 min read Overfitting and Evaluation Design — Be Suspicious of 99% Accuracy Why a model that fits its training data perfectly falls apart in production: the division of labour between train, validation and test, cross-validation, the data leakage that ruins more projects than any modelling mistake, and how accuracy lies on imbalanced data.