Information Theory and AI — Where Cross-Entropy Loss Came From

Information is surprise, entropy is average surprise, KL divergence is the gap between two distributions. Stack those three and the loss function you already use for classification falls out as the only reasonable answer — including why it involves a logarithm, and what perplexity is really counting.

The goal: being able to read this equation

When you train a classifier, this is the quantity actually being minimized.

What it says is: under the true distribution $p$, look at the probability the model's distribution $q$ assigned to each outcome, take its log, average, and negate. $p$ is the distribution on the answer side, $q$ is what the model produced, and $\sum_x$ sums over every outcome that could occur. This is cross-entropy.

Put in words, with the symbols stripped out: the model is graded on how much probability it placed on the things that actually happened, and the grade gets worse the more probability it withheld from them. Nothing in the formula asks the model to be right; it asks the model to have expected what came.

There is one question worth asking. Why this shape? In Probability and Statistics for AI we arrived at the same formula through maximum likelihood. This article walks in from a completely different entrance — the question of what information is — and comes out in the same place. Two independent roads meeting at one point is the strongest evidence that this loss function is not arbitrary.

Information: the size of a surprise

What would it mean to measure information? Shannon's 1948 answer was ruthlessly practical. The amount of information in a message is how surprised you are to hear it.

"The sun will rise in the east tomorrow" — zero information. "It will snow in Tokyo tomorrow" — in August, enormous. The less likely something was, the more it carries when it happens.

To turn that into a formula, the function needs three properties. It should be zero for a certain event. It should grow as probability shrinks. And, decisively, hearing about two independent events together should give the sum of their individual amounts. That third one settles everything: probabilities compose by multiplication, but information ought to compose by addition, and exactly one family of functions turns multiplication into addition.

In plain terms: take the log of the probability and flip the sign. Probability 1 gives $-\log 1 = 0$, no surprise at all. Probability 0.5 gives about 0.69, probability 0.01 about 4.6, and as probability approaches zero it runs off to infinity. The logarithm is not convention or taste — it is forced by the requirement that probabilities multiply while information adds.

Read as an English sentence, it is a definition which says: the more confidently you would have ruled something out, the more you learn when it happens anyway. The news that a coin came up heads barely moves you; the news that a specific lottery ticket won moves you enormously.

The base of the log fixes the unit: base 2 gives bits, the natural log gives nats. The loss values your framework prints are, almost without exception, in nats.

Entropy: average surprise

With the surprise of a single outcome defined, the next question is how unpredictable the distribution is as a whole. Average the individual surprises, weighted by how often each occurs.

What this says is: draw once from this distribution — how surprised will you be, on average? Which is another way of saying how hard the distribution is to predict.

Stated in words: entropy is how much you should expect to learn from a single draw, and that is the same as how badly you would do trying to call the result in advance. A distribution you can already guess teaches you nothing when you look; a distribution you cannot guess teaches you a lot.

Check it against extremes. A rigged coin that always lands heads never surprises you: entropy zero. A fair coin: one bit. A fair six-sided die: about 2.58 bits. More options, spread more evenly, means higher entropy; concentration on a single outcome means lower.