Chapter 0-3 showed that feeding a Bernoulli distribution into maximum likelihood produces cross-entropy. As a formula it was finished there. What is left is the name. Why "entropy"? Why "cross"?

Recovering a name would be trivia on its own, but this chapter buys something else: how far the loss can fall. Chapter 0-2 showed that a minimum need not exist. Here we see the opposite case — a floor above zero. Its height is exactly the entropy.

1. Information — the size of a surprise

"The sun will rise tomorrow" tells you nothing. "It will snow tomorrow" tells you something. The less likely it was, the more there is in being told. Write that as a number.

Definition. The information (self-information) of an outcome $x$ occurring with probability $p(x)$ is

Why a logarithm is forced by three properties you would want:

1. $p = 1$ (certain) carries information $0$
2. smaller $p$ carries more (monotone decreasing)
3. being told about two independent outcomes together carries the sum of being told separately

The third does the work. For independent outcomes $p(x, y) = p(x)p(y)$, so $I$ must turn multiplication into addition — and the only continuous functions that do are constant multiples of $\log$. The constant is the unit: base 2 gives bits, base $e$ gives nats. Deep learning implementations use $e$ almost everywhere (log is the natural log), so the unit there is nats, but this chapter uses bits, where intuition lives. $1\ \text{nat} = 1.442695\ \text{bits}$.

2. Entropy — average surprise

Definition. The entropy of a distribution $p$ is

Equation (2) is the expectation of equation (1) (Chapter 0-3, equation (3)): draw one sample from this distribution, and how surprised are you on average?

Worked numbers. One coin.

A fair coin is exactly 1 bit — that is what defines the bit: one yes/no question's worth. Bias it and the number falls; make it certain and it is zero, because there is nothing to be surprised by.

The uniform distribution is the maximum. With $K$ equally likely outcomes, $H = \log_2 K$: $3.321928$ bits for $K = 10$, $9.965784$ for $K = 1000$.

3. Cross-entropy — measuring while believing the wrong distribution

Suppose reality is $p$ but you built your code believing it was $q$. Lengths follow $q$, giving $-\log q(x)$, while the frequencies that actually arrive follow $p$. The average length is

That is cross-entropy. The distribution you average over and the distribution that sets the lengths disagree — they cross — hence the name. It is the same expression as equation (7) of Chapter 0-3; that one came from maximum likelihood, this one from code lengths.

Worked numbers. For $p = (0.7,\,0.2,\,0.1)$ and $q = (0.5,\,0.3,\,0.2)$: $H(p) = 1.156780$ and $H(p,q) = 1.279586$ bits. The belief cost you the difference.

4. KL divergence — the part that got longer

Definition. Call that excess

the KL divergence (relative entropy).

For the numbers above, $1.279586 - 1.156780 = 0.122806$ bits; computing the right-hand side of equation (4) directly also gives $0.122806$.

$D_{\mathrm{KL}} \ge 0$, with equality only when $p = q$. Two lines, using $\log x \le x - 1$ for $x > 0$ (equality at $x=1$):

$$-D_{\mathrm{KL}}(p\|q) = \sum_x p(x)\log\frac{q(x)}{p(x)} \;\le\; \sum_x p(x)\left(\frac{q(x)}{p(x)} - 1\right) = \sum_x q(x) - \sum_x p(x) = 1 - 1 = 0$$

so

It is not a distance. The same pair gives different numbers depending on the direction: above, $D_{\mathrm{KL}}(q\|p) = 0.132775$ bits, against $0.122806$. The triangle inequality fails too. Read it not as "how far apart they are" but as "what using $q$ costs you when reality is $p$."

5. The join with training — why this is the loss being minimised

When the label is one-hot. One correct class out of three means $p = (1, 0, 0)$, so $H(p) = 0$, and equation (4) gives

$$H(p,q) = H(p) + D_{\mathrm{KL}}(p\|q) = D_{\mathrm{KL}}(p\|q)$$

the cross-entropy simply is the KL. With $q = (0.5, 0.3, 0.2)$, $H(p,q) = -\log_2 0.5 = 1.000000$ bit $= 0.693147$ nats, and the KL is the same number. So "lower the cross-entropy" reads directly as "move the model's distribution toward the true one." That recovers the name.

When the label is the data itself. In real training the true per-class probabilities are unknown; all you have is $m$ examples. Put mass $1/m$ on each and use that empirical distribution $\hat{p}$ in place of $p$:

The right-hand side is equation (6) of Chapter 0-3 — the mean negative log-likelihood itself. "Minimise cross-entropy against the empirical distribution" and "do maximum likelihood" are one operation under two names. The worked example from that chapter ($y=(1,0,1,1,0)$, $p=(0.9,0.2,0.7,0.6,0.3)$) is $0.31053592$ nats $= 0.448009$ bits.

A common misreading: "the loss won't approach zero, so the model must be too weak"

Equation (5) says $H(p,q) \ge H(p)$ for every $q$, with equality only at $q = p$:

The floor of the loss is $H(p)$, and it has nothing to do with the model. One-hot labels give $H(p) = 0$ and a floor at zero, but whenever the same input can carry different answers — label noise, genuinely ambiguous cases, measurement spread — $H(p) > 0$ lifts the floor off the ground.

Measure it. Suppose for some input the truth is $P(y=1 \mid x) = 0.9$: one time in ten the label is 0.

The best reachable value is $0.325083$ nats $= 0.468996$ bits, and that is the floor. Sweeping $q$ numerically puts the minimum at $q = 0.90000$ with value $0.325083$ — the theory's number exactly.

The last three rows are what bites in practice. Raising confidence moves accuracy by not one point and makes the loss worse. At $q = 0.999$ the accuracy is still 90% while the loss has more than doubled. "Push it to be more confident and the loss will drop" is backwards: being more certain than the truth is punished. That is the entrance to calibration, and when a loss stops falling, the first thing to question is not model capacity but how high the floor was to begin with.

Exercises

The answers are folded away. Work them first.

Exercise 1. With $p = (0.7,\,0.2,\,0.1)$ and $q = (0.5,\,0.3,\,0.2)$: (a) what does the gap between $H(p)$ and $H(p,q)$ represent? (b) Does $D_{\mathrm{KL}}(q\|p)$ equal $D_{\mathrm{KL}}(p\|q)$? (c) Give two reasons KL cannot be called a distance.

Exercise 2. Three-class classification, the true class is class 1, the model outputs $q = (0.5,\,0.3,\,0.2)$. (a) The cross-entropy in bits and nats. (b) What are $H(p)$ and $D_{\mathrm{KL}}(p\|q)$ here? (c) What happens to the loss as $q_1$ rises to $0.7,\,0.9,\,0.99$?

Exercise 3. For some input the truth is $P(y=1\mid x) = 0.8$. (a) The smallest achievable cross-entropy (nats). (b) The accuracy there. (c) What happens if the model outputs $q = 1.0$?

What this chapter does not say

• Mutual information and conditional entropy are not covered. "How much does knowing $X$ reduce the uncertainty in $Y$" is writable with this chapter's tools, but the places it gets used (feature selection, representation learning) come later, so it goes there.
• Entropy of a continuous distribution (differential entropy) is a different object. Turn equation (2) into an integral and it can go negative, and the "average code length" reading breaks. Everything here is discrete.
• Shannon's coding theorem is used, not proved. Its content is both halves: you cannot go below $H(p)$, and you can get arbitrarily close to it.
• Calibration — whether a model's stated probabilities match observed frequencies — is not measured or corrected here. This chapter only shows the shape: overconfidence is punished by the loss.
• Every number in the text was computed and checked (the coin and uniform entropies, that $H(p,q) = H(p) + D_{\mathrm{KL}}$ holds, both directions of the KL, the one-hot case, the floor at $P(y=1)=0.9$ against a numerical sweep of $q$, and that 20,000 random pairs produced no negative $D_{\mathrm{KL}}$).

Going deeper

• Information theory and AI — where the cross-entropy loss came from
• Entropy and cross-entropy — where loss functions come from
• KL divergence from scratch — measuring the gap between two distributions
• Mutual information — measuring "knowing"
• Entropy coding from scratch — from Huffman to arithmetic coding
• Compression = prediction = intelligence — LLMs through information theory