Chapter 0-2 settled "which way makes it better." What is left is what counts as better. The training objective — the loss — did not fall from the sky. Write down one assumption in the language of probability and the loss is decided for you.

Probability shows up in deep learning in three places.

1. The output itself. The 0.87 a classifier returns is a probability: how sure it is.
2. The objective. Cross-entropy and squared error both follow from distributional assumptions (section 5).
3. The gradient itself. A minibatch gradient is an estimate of the full-data gradient, and estimates carry error (section 6).

This chapter lays down exactly what those three need.

1. Sample space, events, three promises

Definition. The set of all possible outcomes is the sample space $\Omega$; its subsets are events. A probability $P(A)$ is a number satisfying:

1. $P(A) \ge 0$
2. $P(\Omega) = 1$
3. if $A \cap B = \varnothing$ (they cannot both happen) then $P(A \cup B) = P(A) + P(B)$

That is all. What probability means — a long-run frequency, a degree of belief — is not in these three. The arithmetic needs only the three promises; the interpretation sits outside it.

The third gives a familiar form at once: $A$ and $A^{c}$ cannot both happen and together make $\Omega$, so $P(A^{c}) = 1 - P(A)$.

2. Conditional probability and Bayes — swapping the direction

Definition. For $P(B) > 0$, the probability of $A$ given that $B$ happened is

Equation (1) swaps the denominator. Stop measuring $A$ inside the whole world $\Omega$ and measure it inside $B$ instead.

Dividing the ends of $P(A \cap B) = P(A \mid B) P(B) = P(B \mid A) P(A)$ by $P(B)$,

Bayes' theorem. The derivation is one line and the content is "the same $P(A \cap B)$ written two ways." It earns its keep because the direction you can measure and the direction you want rarely match. A test gives you $P(+ \mid D)$, the fraction of sick people who test positive; you want $P(D \mid +)$, the fraction of positives who are sick. Equation (2) swaps them, and the swap always costs you a $P(A)$ — how likely it was beforehand. Dropping that term is this chapter's common misreading.

Independence. When $P(A \cap B) = P(A)P(B)$, $A$ and $B$ are independent. Substituting into equation (1) gives $P(A \mid B) = P(A)$: learning $B$ does not move your view of $A$.

3. Random variables, expectation, variance

Attach a number to each outcome and you have a random variable (a die face, an image label, one example's gradient).

Definition. The expectation of a random variable $X$ is

and its variance is $\mathrm{Var}(X) = \mathbb{E}\big[(X - \mathbb{E}[X])^2\big]$.

Expectation is a weighted average. Variance measures spread around that average, and expanding gives

Its square root $\sigma$ is the standard deviation, carried in the same units as $X$.

Worked numbers. For a fair die, $\mathbb{E}[X] = 3.5$, $\mathbb{E}[X^2] = 15.1667$, $\mathrm{Var}(X) = 15.1667 - 12.25 = 2.9167\ (= 35/12)$, $\sigma = 1.7078$.

Only two properties are needed here.

• Linearity. $\mathbb{E}[aX + bY] = a\,\mathbb{E}[X] + b\,\mathbb{E}[Y]$ — this holds even when they are dependent.
• Variances add when independent. If $X, Y$ are independent, $\mathrm{Var}(X + Y) = \mathrm{Var}(X) + \mathrm{Var}(Y)$. This one needs independence.

From the second, for the mean $\bar{X} = \frac{1}{B}\sum_{i=1}^{B} X_i$ of $B$ independent observations,

The centre is right whatever $B$ is; only the spread shrinks, and it shrinks by $\sqrt{B}$. Equation (5) is the workhorse of this chapter and returns in section 6 as minibatching. Note $\sqrt{B}$, not $B$.

4. Three distributions worth knowing

Bernoulli. One shot at 0 or 1. With $p \in [0,1]$, $P(y) = p^{y}(1-p)^{1-y}$ for $y \in \{0,1\}$ — since $y$ is 0 or 1, one exponent is always 0 and that factor disappears. Mean $p$, variance $p(1-p)$. The $\sigma(z)$ that a binary classifier outputs is this $p$.

Categorical. One of $K$. $P(y = k) = p_k$ with $p_k \ge 0$ and $\sum_k p_k = 1$. Softmax is the map that forces numbers into exactly those constraints: $\mathrm{softmax}(z)_k = e^{z_k}/\sum_j e^{z_j}$ makes them positive with the exponential and sums to one by dividing — one step per constraint.

Gaussian (normal). Continuous values.

$$p(y) \;=\; \frac{1}{\sigma\sqrt{2\pi}} \exp\!\left(-\frac{(y-\mu)^2}{2\sigma^2}\right)$$

with mean $\mu$ and variance $\sigma^2$. Assume it in regression and squared error falls out (next section).

5. Maximum likelihood — where the loss came from

Given data $(x_i, y_i)_{i=1}^{m}$ and a model that emits a conditional distribution $p(y \mid x; \theta)$, choose $\theta$ so that the data you actually have is as likely as possible. That is maximum likelihood.

Three things happen at that equals sign. Take logs (products become sums, and the underflow of multiplying $m$ small numbers goes away; $\log$ is increasing so the arg max does not move). Flip the sign (maximise becomes minimise). Divide by $m$ (so the scale does not change with the number of examples).

Now substitute the distributions from section 4 and familiar losses appear.

Substituting Bernoulli. With $p(y \mid x) = p^{y}(1-p)^{1-y}$ and $p = \sigma(z)$, $\log p(y\mid x) = y\log p + (1-y)\log(1-p)$, so equation (6) becomes

Cross-entropy. Chapter 1 of Part III reached the same expression by saying "we want $p_i$ large where the label is 1 and $1-p_i$ large where it is 0; written together in one line." What that "written together" was is the Bernoulli probability itself.

Check it numerically. For $y = (1,0,1,1,0)$ and $p = (0.9,\,0.2,\,0.7,\,0.6,\,0.3)$ the likelihood product is $0.9 \times 0.8 \times 0.7 \times 0.6 \times 0.7 = 0.21168$, and $-\log(0.21168)/5 = 0.31053592$. Computing equation (7) directly also gives $0.31053592$ — identical, to zero difference.

Substituting Gaussian. With $y \mid x \sim \mathcal{N}(\hat{y}, \sigma^2)$, one example's negative log-likelihood is

The second term has no $\hat{y}$ in it. Minimise over $\hat{y}$ and only squared error remains; fixing $\sigma$ leaves nothing but a constant factor. For $y = (2.0,-1.0,0.5)$, $\hat{y} = (1.5,-0.7,0.9)$, $\sigma = 1$, the mean negative log-likelihood is $1.002272$ and the mean squared error $0.166667$, differing by exactly $0.918939 = \log\sqrt{2\pi}$ — a constant, as promised.

6. Minibatches — the gradient is an estimate, not a measurement

One training step's gradient is really an average over all the data:

$$\nabla L = \frac{1}{m}\sum_{i=1}^{m} \nabla \ell_i \;=\; \mathbb{E}\big[\nabla \ell\big]$$

With $m$ in the millions, every single step would read millions of examples. So draw $B$ of them and use their average — that is a minibatch, and equation (5) applies verbatim. The centre is right (unbiased); the spread is $\sigma/\sqrt{B}$.

Measured. Two-dimensional logistic regression, 1,000 examples (seed 20261013), evaluated at $w = (0.5, -0.3)$, $b = 0.1$: I formed the per-example gradient $\partial \ell_i/\partial w_1$ for all of them. The full-data gradient is $-0.083919$ and the per-example standard deviation is $\sigma = 0.375821$. Drawing $B$ of them and averaging, 20,000 times for each $B$:

At $B = 1$ the error, $0.372$, is more than four times the gradient itself, $0.084$. Steps that point the wrong way are routine. Training still works because the estimate is unbiased: average enough of them and they pull the right way. The first half of equation (5) is that guarantee — and nothing guarantees any individual step.

Out of this comes an operational number. Four times the batch costs four times the compute and halves the error. Going $B = 16 \to 64$ takes the error from $0.0928$ to $0.0469$, for four times the work. That is why bigger batches are not simply better.

A common misreading: "a 99%-accurate test came back positive, so I'm 99% likely to be ill"

Drop the $P(A)$ in equation (2) — how likely it was beforehand — and this is what you get. The numbers make the gap plain.

Take a disease with prevalence 0.1%, a test with sensitivity (sick people who test positive) 99% and specificity (healthy people who test negative) 99%. Count out 10,000 people.

$9.9 + 99.9 = 109.8$ people test positive, of whom $9.9$ are actually ill.

$$P(D \mid +) \;=\; \frac{9.9}{109.8} \;=\; 9.0164\%$$

Nine per cent, not ninety-nine. The test did nothing wrong. There are 9,990 healthy people, so even a 1% error rate produces 99.9 of them, swamping the 9.9 real cases. In equation (2), the small factor $P(D) = 0.001$ multiplies the numerator only.

Change the prevalence and the answer changes, with the test untouched.

At 1% it is exactly a coin flip; only at 50% does it reach 99%.

Exercises

The answers are folded away. Work them first.

Exercise 1. Apply the same test (sensitivity 99%, specificity 99%) to a population with prevalence 10%. (a) What is $P(D \mid +)$? (b) Why does the answer change when the test did not? (c) What is the classifier equivalent of this story?

Exercise 2. At some point in training, the per-example gradient has standard deviation $\sigma = 0.4$. (a) The standard error at $B = 16$. (b) What $B$ halves it? (c) What $B$ divides it by ten? (d) What does that tell you?

Exercise 3. Regression assuming $y \mid x \sim \mathcal{N}(\hat y, \sigma^2)$. (a) Write one example's negative log-likelihood and point out the term with no $\hat y$ in it. (b) Does doubling $\sigma$ change the minimising $\hat y$? (c) What happens when the errors are not Gaussian but include outliers?

What this chapter does not say

• Entropy and KL divergence belong to the next chapter (0-4). This chapter stops at maximising likelihood; where the name "cross-entropy" comes from — its meaning as a quantity in information theory — is not touched. The formula itself is fully derived in section 5.
• Bayesian inference (carrying a posterior around) is not covered. Equation (2) was used as a one-off calculation; putting priors on parameters and training distributions is a different programme.
• Continuous random variables are not treated rigorously (measure, and how a density differs from a probability). A density is not a probability and can exceed 1; nothing computed here depends on that distinction.
• The central limit theorem is shown in the figure, not stated or proved. Equation (5) — variance $\sigma^2/B$ — needs only independence; "it goes bell-shaped" is a separate claim.
• The minibatch measurement is one point in parameter space and one component of the gradient. As training moves, $\sigma$ moves too. What does not move is the $1/\sqrt{B}$ shape.
• Every number in the text was computed and checked (the die's variance, the Bayes table and all four prevalences, the five standard errors, the likelihood product against cross-entropy, and the gap between Gaussian negative log-likelihood and squared error being exactly $\log\sqrt{2\pi}$).

Going deeper

• Probability and statistics for AI — a model's output is a probability
• Bayes' theorem from scratch — why a 99%-accurate test becomes a coin flip
• A zoo of distributions — where normal, Poisson and the exponential family come from
• Monte Carlo from scratch — solving integrals with dice
• Statistical learning theory — why learning generalises at all