Ask someone to write down, in words, how to tell a cat from a dog. They stall. Pointed ears, whiskers, vertical pupils — every rule has counter-examples. And yet people recognise a cat instantly. The rule cannot be written, but the judgement works. That gap is where this book starts.

There are two ways forward. Write the rule, or fit the rule.

Writing it: the limit is the number of knobs

The figure opens on a deliberately bad line at 46.7%. Work the sliders patiently and you reach 93.3% — that is the ceiling, measured by exhaustively searching both knobs on this figure's default points. What remains sits where the two clusters overlap; no single straight line can pick it up.

The thing to notice is not that you got there, but why two knobs were enough. The points live on a plane: there are exactly two features, horizontal and vertical.

With $d$ features the formula contains $d+1$ numbers: $d$ weights $w_1,\dots,w_d$ for the orientation and one offset $b$ for the position. Three features make the line a plane, so four numbers; with 784 features — a small 28×28 image, pixels laid out flat — 785. At 785 numbers to turn by hand, there is no human strategy left.

Notice that one number is left over. This figure has $d=2$, so there are three numbers ($w_1, w_2, b$) but only two knobs. Scale $w$ and $b$ together by two and the set of points satisfying $w^{\top}x + b = 0$ is unchanged — the line does not move a millimetre. Only two things are worth moving: direction and position. Angle and offset are those two put directly under your fingers.

There are $d+1$ numbers, but the boundary uses only $d$ of them. The leftover one — the scale — changes something else instead of the boundary. What it changes is measured later in this chapter.

That is the limit of writing rules. It is set by dimension, not by cleverness.

Fitting it: the procedure does not grow with dimension

Switch the figure to "fit it from data". The knobs lock and the line starts moving on its own. It settles at 91.7% — just under the 93.3% ceiling you can reach by hand, both measured on the same points. It got there without a single knob being touched.

What it does is one line. Judge the points with the current line, then nudge the line in the direction of how it was wrong. Repeat.

Here $\sigma(z) = 1/(1+e^{-z})$, $y_i$ is the label (0 or 1) and $\eta$ is the step size. The part to watch in equation (1) is $(\sigma(\cdot) - y_i)$: the gap between prediction and truth. Zero gap, no movement; large gap, large movement.

What matters most is that the equation does not mention the dimension of $x$. Two features or 784, the formula is identical — only the loop count changes. The "fit" side of the figure is running exactly this, one step per frame.

(The fitted line does not quite reach 93.3%, and that is not sloppiness. Equation (1) lowers log loss, not accuracy; they are different quantities. The numbers are below.)

Writing becomes unmanageable as dimension grows; fitting does not change shape at all. That asymmetry is the reason to use machine learning.

Where equation (1) comes from

"Nudge it in the direction of how it was wrong" was a paraphrase, but nobody picked that direction by hand. Fix one quantity you want to lower and the direction follows uniquely from a derivative. Equation (1) is that derivative. Here is the quantity.

Read $p_i = \sigma(w^{\top}x_i + b)$ as "the probability that this point has $y_i=1$". Where the label is 1 you want $p_i$ large; where it is 0 you want $1-p_i$ large. Both at once is $p_i^{\,y_i}(1-p_i)^{1-y_i}$ — since $y_i$ is 0 or 1, one factor always has exponent 0 and disappears. Wanting it large for all $m$ points at once means a product; a product becomes a sum under a logarithm; and "make it large" becomes "make it small" by flipping the sign.

This is cross-entropy (log loss). At a point with $y_i=1$ only $-\log p_i$ survives: it goes to zero as $p_i$ approaches 1, and grows without bound as $p_i$ approaches 0. Right and confident is cheap; wrong and confident is enormous. Accuracy has no such notion of how badly you were wrong — it counts one vote per point.

Now differentiate $L$ with respect to $w$. Three steps.

Step 1. The logistic function's derivative is written with itself: $\sigma'(z) = \sigma(z)\big(1-\sigma(z)\big) = p(1-p)$.

Step 2. Differentiate one point's loss $\ell = -\big[y\log p + (1-y)\log(1-p)\big]$ with respect to $z = w^{\top}x + b$. By the chain rule $\partial \ell/\partial z = (\partial \ell/\partial p)\cdot(\partial p/\partial z)$, where

$$\frac{\partial \ell}{\partial p} = -\left(\frac{y}{p} - \frac{1-y}{1-p}\right), \qquad \frac{\partial p}{\partial z} = p(1-p)$$

Multiplying clears the denominators:

$$\frac{\partial \ell}{\partial z} = -\big[\, y(1-p) - (1-y)p \,\big] = -\big[\, y - yp - p + yp \,\big] = p - y$$

That $p(1-p)$ cancels is itself the reason to choose this loss. What is left is $p - y$ — the gap between prediction and truth, the thing inside equation (1)'s brackets.

Step 3. Since $z = w^{\top}x + b$, we have $\partial z/\partial w = x$ and $\partial z/\partial b = 1$. Averaging over $m$ points:

So equation (1) is $w \leftarrow w - \eta\,\partial L/\partial w$: one step in the direction that lowers the loss, and nothing more. (Equation (3) was checked against numerical differentiation on this chapter's own 60 points; the largest discrepancy is $1.6\times10^{-10}$.)

The reason to go deep has not appeared yet

So far there is one straight line. For arrangements a line cannot split — one class enclosing the other, say — no angle wins. Push the figure's "spread" up and the clusters merge and accuracy falls. That is not a badly drawn line; the shape "straight line" is insufficient.

Two options follow.

1. Build features by hand. Add a column such as "distance from the centre" and a straight line can separate it again. This was the mainstream approach for a long time, and designing those columns was the craft.
2. Fit the features too. Learn the stage that builds features from the input together with the stage that classifies, under the same equation.

The second is what "going deep" means. Layers are stacked not so that a human can assign meaning to each one, but so that the work of building features is also handed to equation (1). The remaining eleven chapters are, almost entirely, the consequences of that one move.

• Chapter 2: making the fit stronger makes it fit only the data you hold (overfitting).
• Chapter 3: with layers stacked, how the "gap" in equation (1) reaches the lower layers. The chain rule is all it takes.
• Chapter 4: choosing $\eta$ and treating overfitting turn out to be two sides of the same question.
• Chapter 5 onward: what assumption to build into the structure, according to the shape of the data.

A common misconception: "accuracy up means loss down"

They move together often enough to look like one quantity. They are two, and this chapter's figure pulls them apart.

The source of the gap is the number we left over. Accuracy only reads the sign of $w^{\top}x + b$, so scaling $w$ and $b$ by the same positive factor changes nothing at all. Loss reads the value of $p = \sigma(w^{\top}x + b)$, so scaling pushes $p$ towards 0 or 1 — cheaper at the points you get right, far more expensive at the points you get wrong. Scale changes the loss without changing the boundary. Measured on the figure's default 60 points:

Rows 1 and 2 are the same boundary. Not one point is classified differently, yet the loss is 0.4838 against 0.2895 — nearly double. Accuracy looks only at the boundary; loss looks at the boundary and at the confidence. That is what "different quantities" means.

It breaks the other way too. Across those 240 steps there were three occasions where the loss went down and the accuracy went down with it: step 1→2, accuracy 88.3%→81.7% with loss 0.6848→0.6775; step 25→26, 88.3%→86.7% with 0.5547→0.5509; step 35→36, 86.7%→85.0% with 0.5205→0.5175. Equation (1) lowered the loss every single time, and accuracy still fell.

So the fitted side stopping at 91.7%, short of the hand-made 93.3%, is not a question of skill. Equation (1) is doing its job — lowering loss — and that job is not "maximise accuracy". By loss, row 3 (0.3420) beats row 1 (0.4838); but row 2 shows the hand-made boundary reaches 0.2895 if you pick its scale. In other words the fitted side has not reached the bottom of the loss either at 240 steps. Where to stop is chapter 4.

Exercises

The answers are folded away. Work them out first.

Exercise 1. For one point's loss $\ell = -\big[y\log\sigma(z) + (1-y)\log(1-\sigma(z))\big]$, show that $\partial\ell/\partial z = \sigma(z) - 1$ in the case $y=1$. You may use $\sigma'(z) = \sigma(z)(1-\sigma(z))$.

Exercise 2. Suppose you feed 28×28 pixels in directly as features. (a) How many numbers does equation (1) move? (b) How many of them determine the shape of the boundary? (c) What does the remaining one change?

Exercise 3. Three points, all with label $y=1$. Model A predicts $p=(0.55,\,0.55,\,0.05)$; model B predicts $p=(0.45,\,0.45,\,0.45)$. Call it 1 when $p>0.5$. (a) The accuracy of each. (b) The mean log loss of each. (c) Does "higher accuracy means lower loss" hold?

What this chapter does not claim

• It does not say deeper is always better. Adding layers destabilises training and needs the treatments in chapter 4. Where data is scarce, hand-built features still win.
• Equation (1) is the simplest classifier (logistic regression). Cross-entropy is derived here, but why it is the right loss to pick — its relation to maximum likelihood, and how it compares with other choices — is chapter 2.
• The numbers in "a common misconception" are measurements on this chapter's default points (60 synthetic points). Change the count or the spread and the values change. What does not change is that accuracy is blind to scale and loss is sensitive to it.
• The figure's points are synthetic. Properties of real data — missingness, bias, drift — are not covered.

Going deeper

This chapter is the map. The detail is in the articles.

• What machine learning is — understanding "learning" without equations
• Loss and optimization — how a model learns from being wrong
• Neural networks from scratch — from one neuron to many layers