Information Theory and AI — Where Cross-Entropy Loss Came From

Information is surprise, entropy is average surprise, KL divergence is the gap between two distributions. Stack those three and the loss function you already use for classification falls out as the only reasonable answer — including why it involves a logarithm, and what perplexity is really counting.

The goal: being able to read this equation

When you train a classifier, this is the quantity actually being minimized.

What it says is: under the true distribution $p$, look at the probability the model's distribution $q$ assigned to each outcome, take its log, average, and negate. $p$ is the distribution on the answer side, $q$ is what the model produced, and $\sum_x$ sums over every outcome that could occur. This is cross-entropy.

Put in words, with the symbols stripped out: the model is graded on how much probability it placed on the things that actually happened, and the grade gets worse the more probability it withheld from them. Nothing in the formula asks the model to be right; it asks the model to have expected what came.

There is one question worth asking. Why this shape? In Probability and Statistics for AI we arrived at the same formula through maximum likelihood. This article walks in from a completely different entrance — the question of what information is — and comes out in the same place. Two independent roads meeting at one point is the strongest evidence that this loss function is not arbitrary.

Information: the size of a surprise

What would it mean to measure information? Shannon's 1948 answer was ruthlessly practical. The amount of information in a message is how surprised you are to hear it.

"The sun will rise in the east tomorrow" — zero information. "It will snow in Tokyo tomorrow" — in August, enormous. The less likely something was, the more it carries when it happens.

To turn that into a formula, the function needs three properties. It should be zero for a certain event. It should grow as probability shrinks. And, decisively, hearing about two independent events together should give the sum of their individual amounts. That third one settles everything: probabilities compose by multiplication, but information ought to compose by addition, and exactly one family of functions turns multiplication into addition.

In plain terms: take the log of the probability and flip the sign. Probability 1 gives $-\log 1 = 0$, no surprise at all. Probability 0.5 gives about 0.69, probability 0.01 about 4.6, and as probability approaches zero it runs off to infinity. The logarithm is not convention or taste — it is forced by the requirement that probabilities multiply while information adds.

Read as an English sentence, it is a definition which says: the more confidently you would have ruled something out, the more you learn when it happens anyway. The news that a coin came up heads barely moves you; the news that a specific lottery ticket won moves you enormously.

The base of the log fixes the unit: base 2 gives bits, the natural log gives nats. The loss values your framework prints are, almost without exception, in nats.

Entropy: average surprise

With the surprise of a single outcome defined, the next question is how unpredictable the distribution is as a whole. Average the individual surprises, weighted by how often each occurs.

What this says is: draw once from this distribution — how surprised will you be, on average? Which is another way of saying how hard the distribution is to predict.

Stated in words: entropy is how much you should expect to learn from a single draw, and that is the same as how badly you would do trying to call the result in advance. A distribution you can already guess teaches you nothing when you look; a distribution you cannot guess teaches you a lot.

Check it against extremes. A rigged coin that always lands heads never surprises you: entropy zero. A fair coin: one bit. A fair six-sided die: about 2.58 bits. More options, spread more evenly, means higher entropy; concentration on a single outcome means lower.

KL divergence: the gap between two distributions

Now the real work. Reality follows $p$, but we believe it follows $q$. What does that mistake cost?

What this says is: when events actually unfold according to $p$, this is the extra surprise you pay on average for having believed $q$. The term $\log(p/q)$ measures, on a log scale, the discrepancy between the real probability and the believed one; it is then averaged with weights given by how often reality produces each outcome.

In one sentence, it is a number which says how much extra you get startled, day after day, because your picture of the world is not the world. Note where the weights come from: the mistakes that count are the ones you make about things that actually happen often. Being badly wrong about an outcome that never occurs costs you nothing.

Three properties are worth memorising.

• It is never negative. Being wrong never pays.
• It is zero only when $p = q$. No gap, no cost.
• It is not symmetric. $D_{\mathrm{KL}}(p\|q)$ and $D_{\mathrm{KL}}(q\|p)$ are different quantities, which is why it is a divergence and not a distance.

That third point has teeth. Minimizing $D_{\mathrm{KL}}(p\|q)$ punishes $q$ severely for putting near-zero probability anywhere $p$ has mass, since $\log(p/0)$ blows up — so $q$ spreads out to cover everything, blurry but omitting nothing. Minimizing the reverse, $D_{\mathrm{KL}}(q\|p)$, lets $q$ settle comfortably onto one peak of $p$: sharp, but narrow.

KL also shows up throughout training itself. Knowledge distillation shrinks the KL between a teacher's distribution and a student's. RLHF-style methods add a KL penalty against a reference model to stop the policy drifting too far from where it started. KL is the common yardstick for "how different are these two distributions allowed to be".

Cross-entropy equals entropy plus KL

All three quantities are now on the table, and they connect in one clean line.

What this says is: the total surprise the model pays is the unavoidable uncertainty plus the surcharge for being wrong.

It is an accounting identity which says the bill has exactly two lines: one you were always going to pay because the data is genuinely uncertain, and one you are paying only because your model is off. Training can touch the second line and can never touch the first.

And here is the decisive fact. $H(p)$ is a property of the data alone; it does not depend on the model's parameters at all. It is a constant. Therefore:

Minimizing cross-entropy is exactly, identically, minimizing KL divergence.

Driving the loss down was never anything other than pulling the model's distribution toward the data's distribution. When the training curve descends, the part descending is the KL term; $H(p)$ never moves. Which means the loss has a floor it can never break through, and the height of that floor is $H(p)$ — the irreducible uncertainty in the data itself. A language model's loss does not reach zero because natural language is genuinely ambiguous, not because the implementation is lacking.

Collapse it onto a one-hot label and you get the familiar formula

Bring the general statement down to an actual classification problem. Training labels are usually definite: the answer is class 3. In that case $p$ puts 1 on the correct class and 0 everywhere else — a one-hot distribution.

$$H(p, q) = -\sum_{c} y_c \log q_c = -\log q_{c^*}$$

$y_c$ is the label (1 for the correct class, 0 otherwise), $q_c$ is the probability the model gave class $c$, and $c^*$ is the correct class.

In plain terms: every term with $y_c = 0$ vanishes from the sum, leaving exactly one survivor — minus the log of the probability assigned to the right answer.

Said in words, the whole loss reduces to one question: how much probability did you give the answer that turned out to be correct? What the model said about the wrong classes never appears in the formula — it matters only because probability spent there is probability taken from the right one.

That is precisely the $-\log p_i$ handed down without justification as "the default for classification" in Loss Functions and Optimization. It is also, character for character, the negative log-likelihood derived from maximum likelihood in the previous article. Walk in from information theory or walk in from probability, and you land on the line of code that is already in your training script. That, not habit, is why cross-entropy is the standard.

Keep a feel for the numbers too. Assign 0.9 to the truth and you pay almost nothing; assign 0.01 and the penalty jumps to about 4.6. That brutal hatred of confident errors was not designed in either — it falls straight out of defining information as surprise.

Perplexity: how many options are you effectively torn between?

Perplexity, the number quoted in every language modelling result, sits on the same line of reasoning.

$T$ is the token count and $p_{\theta}(x_t \mid x_{<t})$ is the probability the model gave to the token that actually appeared, given the preceding context. The expression inside the exponential is cross-entropy per token, nothing else.

What this says is: take the average surprise and exponentiate it back into a count of options. Entropy lives in log-space, so undoing the log restores a sense of "how many". Which leaves a question anyone can picture: if the model produced each next token by picking blindly off a shortlist, how long would that shortlist have to be to explain the performance you are seeing? A perplexity of 20 means the model is, on average, choosing among roughly twenty plausible next tokens. A perplexity of 1 is perfect prediction; a perplexity equal to the vocabulary size means nothing was learned.

One caution matters in practice. Perplexities from models with different tokenizers or evaluation sets cannot be compared. The same passage split differently yields a different token count, and the per-token value moves accordingly. Check that the premises match before lining numbers up.

Three things that matter in practice

1. Know what unit your loss is in The cross-entropy your framework prints is in nats. Divide by $\log 2 \approx 0.693$ for bits, exponentiate for perplexity. A loss of 1.6 becomes $e^{1.6} \approx 5$ — the model is effectively picking among about five options.

2. Label smoothing is "flatten the target distribution slightly" Instead of demanding 1.0 on the correct class, put 0.9 there and spread the remainder thinly across the others. A one-hot target demands that the correct probability reach 1, which is to say that its logit reach infinity, and that pressure encourages overconfidence. Nudging the target's entropy up slightly relieves it.

3. When you see a KL, check its direction Whenever $D_{\mathrm{KL}}$ appears in a paper, look first at which distribution is the first argument. As above, the direction decides whether the behaviour is "cover everything" or "commit to one mode" — and it is a reliable source of implementation bugs.

Summary

• Information $-\log p(x)$ is the size of a surprise; the logarithm is forced by wanting probabilities to multiply while information adds
• Entropy $H(p)$ is average surprise: large for flat distributions, small for concentrated ones
• KL divergence is the extra surprise paid for believing $q$ when reality is $p$ — non-negative, asymmetric, and not a distance
• Cross-entropy equals entropy plus KL; since $H(p)$ is constant, minimizing the loss is minimizing KL, and $H(p)$ is the floor the loss can never break
• With a one-hot label, cross-entropy collapses to minus the log of the probability given to the right answer — identical to the maximum likelihood result
• Perplexity is cross-entropy exponentiated back into a count of effective options

Now that the origin of the loss is settled, the remaining question is how to push it down. That machinery is in Loss Functions and Optimization and Backpropagation From Scratch.