Entropy and Cross-Entropy — Where the Loss Function Comes From

The cross-entropy loss you write in every classifier wasn't handed down by decree. Starting from the idea of surprise, we walk a single path through entropy and cross-entropy to that familiar one-liner — and finish with why squared error is the wrong tool for the job.

Nothing obvious carries information

Imagine a desert town where it almost never rains. The forecast says "sunny again tomorrow." You have learned nothing — you already knew. Say instead "snow tomorrow," and it's front-page news.

That's what information actually is: the amount of information in an event is set by how surprising it was. What Claude Shannon formalized in 1948 was a way to put a number on that surprise.

It looks like a detour, but this is exactly where the cross-entropy loss used in every image classifier and every language model comes from. Here's the path from surprise to that equation, with no gaps.

Turning surprise into a formula — why a logarithm?

We want a function that measures surprise. It only needs two properties:

1. The less likely the event, the bigger the surprise
2. When two independent events both happen, the surprise should be the sum of the individual surprises

The second one settles it. Two independent events both happening is a product of probabilities ($p_1 \times p_2$). A function that turns products into sums is the logarithm — and up to a constant factor, it's the only function satisfying both conditions.

Equation (1) says: the information in event $x$ is the log of one over its probability $p(x)$. Every time the probability halves, the information goes up by a fixed amount.

Put in words: hearing about a one-in-a-hundred event hands you far more than hearing that a coin came up heads. Rarity is converted straight into quantity.

The base of the log sets the unit. Base 2 gives bits, the natural log gives nats. Learning the outcome of one coin flip is $\log_2 2 = 1$ bit; two flips is 2 bits. "One bit = one binary choice resolved" is a fair reading. ML libraries use the natural log, so your loss values are in nats.

Entropy: surprise on average

Instead of one event, we want to grade the source. So take the average surprise.

Equation (2) lines up every possible outcome, multiplies each one's surprise $\log\frac{1}{p(x)}$ by how often it actually happens, $p(x)$, and adds them up. It answers: how surprising is this source, on average?

Said in words, it is the surprise bill this source charges you per event, averaged over a long run of events.

A fair coin gives $H = 1$ bit. A coin that lands heads 90% of the time gives about 0.47 bits. A coin that's essentially always heads approaches 0. The more predictable the source, the lower its entropy. The maximum comes when every outcome is equally likely ($\log N$ for $N$ options) — the least predictable state there is.

Entropy also has an engineering meaning: it is the theoretical floor on the average number of bits needed to record that source. Give short codes to frequent symbols and your average length shrinks, but no coding scheme, however clever, gets below this number.

Cross-entropy: the cost of living by the wrong forecast

Now the real question. We never know the true distribution $p$. All we have is the model's guess, $q$.

Picture yourself building a codebook out of $q$: "sunny is common, give it a short code; snow is rare, give it a long one." But the weather that actually falls follows $p$. So the length of each code is decided by $q$, while how often you use it is decided by $p$ — a mismatched ledger.

Equation (3) is that average cost, and it is called cross-entropy. It differs from equation (2) in exactly one place: the thing inside the log went from $p$ to $q$. Read it as "the size of each surprise is measured by the model's guess $q$, but how often that surprise shows up is decided by reality $p$."

The plain version, which says the same thing: you handed snow a long codeword and then moved somewhere it snows constantly, and this is the inflated bill you now pay every month. You get to write the codebook out of your own beliefs; the weather decides how many times it gets used.

Split it open and KL divergence falls out

Cross-entropy factors cleanly into two pieces.

Equation (4) reads: the cost of using the wrong forecast = the part nobody can avoid + the surcharge for being wrong. The first term $H(p)$ is how unpredictable reality itself is — a constant the model cannot touch. The second, $D_{\mathrm{KL}}(p\|q)$, is the KL divergence: the penalty for $q$ drifting from $p$, always at least 0, and exactly 0 only when they match.

Put in words, the loss number on your screen already has the difficulty of the data itself folded into it. Training can only eat away at the second term; the first one is still there when you stop.

This split is the justification for using cross-entropy as a loss. If $H(p)$ is constant, then driving cross-entropy down is identical to driving KL divergence down. We aren't minimizing something that merely looks reasonable — we're shrinking the gap between distributions directly. KL's own personality, especially the asymmetry that makes it not a distance, is covered in KL Divergence From Scratch.

Equation (4) also carries a practical warning: the floor of the loss is $H(p)$, not 0. With noisy labels, even a perfect model stops there. When your training curve flattens at 0.3, whether that's the model's ceiling or the data's depends on knowing this term exists.

Collapse it with a one-hot label and you get the familiar line

In classification, exactly one answer is correct. "This image is a cat" means the true distribution $p$ is a one-hot vector: 1 on cat, 0 everywhere else.

Drop that into equation (3) and every term except the correct label $y$ is multiplied by $p(x)=0$ and vanishes. One term survives.

Equation (5) is the cross-entropy loss you actually write. Bluntly read: take the probability the model assigned to the correct label, take its log, flip the sign.

Stated in words: the grader looks at exactly one cell. Once $q_y$ is fixed, how the leftover probability gets divided among the wrong classes makes no difference to the loss at all.

Assign 0.9 to the right answer and the loss is about 0.105; 0.5 gives 0.693; 0.01 gives 4.6. Being confidently wrong is punished sharply — that's this function's whole character. Exponentiating the loss, $e^{L}$, gives perplexity: how many equally-likely options the model is effectively torn between. A loss of 4.6 means roughly a 100-way coin flip.

Why it always comes paired with softmax

A model's final layer doesn't emit probabilities; it emits logits — unbounded raw scores like cat: 3.2, dog: 1.1, bird: -0.5. To feed equation (5) we must convert them, and softmax is what does it.

$$q_i = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}$$

All it does is push each score through an exponential, then divide by the total so the numbers become proportions. The exponential makes negatives positive while preserving the ordering and exaggerating gaps. $T$ is the temperature, 1 by default: raise it and the distribution flattens, lower it and it sharpens.

It is, in words, a translation of a gap in points into a gap in probability, with the ranking left exactly as it was — a scoreboard rewritten as shares of the whole.

The thing to check by dragging the slider: identical scores can produce wildly different losses. If the correct class is barely ahead, cooling the distribution pushes $q_y$ toward 1 and the loss toward 0; heating it pushes $q_y$ toward $1/N$ and pins the loss at $\log N$. Training almost always holds the temperature at 1 — you touch it at inference time to tune diversity, and in knowledge distillation.

Writing it in code

In numpy it's startlingly short.

import numpy as np

def cross_entropy(logits, labels):        # logits: (B, C), labels: (B,)
    z = logits - logits.max(-1, keepdims=True)      # guard against overflow
    logZ = np.log(np.exp(z).sum(-1, keepdims=True)) # log-sum-exp
    logq = z - logZ                                 # = log softmax(logits)
    return -logq[np.arange(len(labels)), labels].mean()

The rule here is: never compute softmax and then take its log. If $q_y$ underflows to 0, $\log 0 = -\infty$ and everything downstream is ruined. Subtract the max, use log-sum-exp, and you get $\log q_y$ directly without ever passing through the dangerous intermediate probability.

Why squared error is the wrong tool

Squared error (MSE) does technically train a classifier — it's just visibly slower. The reason lives in the shape of the gradient.

Pair softmax with cross-entropy and the gradient with respect to the logits becomes a single line.

Equation (6) says: the predicted probabilities minus the one-hot target, and nothing else. Give the right answer only 0.01 and the gradient magnitude is nearly 1 — the more wrong you are, the harder you get corrected.

A sentence which says the same thing: take back what you over-predicted, hand it to what you under-predicted, and stop. The size of the correction is the size of the mistake — that is the whole trick of this pairing.

Switch to MSE and the derivative of softmax (or sigmoid) gets multiplied in on top. That derivative is nearly 0 wherever the output sits near 0 or 1, which produces the worst possible behavior: the gradient vanishes precisely when the model is confidently wrong. Cross-entropy's logarithm is built to cancel that saturating factor exactly. For a refresher on gradient descent itself, see Loss Functions and Optimization.

How this shows up on the job

Who touches it, and when. Any ML engineer training a classifier touches this every single run. People doing continued pretraining or fine-tuning on LLMs read the loss value itself as a monitoring signal, usually converted to perplexity.

The parameter names you'll actually type. nn.CrossEntropyLoss in PyTorch, SparseCategoricalCrossentropy(from_logits=True) in TensorFlow/Keras. Three arguments come up constantly:

• weight (class_weight in Keras): handles class imbalance by counting minority-class loss more heavily
• ignore_index: excludes padding positions in sequence tasks. Forget it and the model diligently learns to emit padding
• label_smoothing: softens the target from 1.0 to around 0.9 (0.1 is the standard setting), reining in overconfidence

Pitfalls that turn into incidents.

1. Double softmax. Leaving a softmax on the final layer and then passing it to nn.CrossEntropyLoss is the single most common bug here. That function expects logits and applies log-softmax internally, so the squashing happens twice, gradients go flat, and it surfaces as the hard-to-spot symptom "training progresses but accuracy plateaus early." Getting from_logits wrong in Keras is the same accident.
2. Don't target a loss of 0. The floor is $H(p)$. If loss pins to 0 on noisy data, suspect memorization rather than generalization.
3. Don't compare loss across tasks. Different class counts mean different baselines. Random guessing on 10 classes is $\log 10 \approx 2.30$; on a 50k vocabulary it's about 10.8. Compare against $\log N$ first.
4. Don't trust the output probabilities blindly. Deep models tend to be overconfident; a 0.99 doesn't guarantee 99% accuracy in practice. If you're automating decisions with a threshold, check calibration on held-out data.

How it gets asked in interviews. For "what's the relationship between cross-entropy and KL divergence," write equation (4) and say the difference is the constant $H(p)$, so the two are equivalent as minimization problems. For "why not MSE for classification," bring out equation (6) and the saturation story.

Summary

• Information is surprise, $-\log p$; entropy is its average, and predictable sources have low entropy
• Cross-entropy is the average cost of a codebook built from the model's guess $q$ but used at reality's frequencies $p$
• Since $H(p,q) = H(p) + D_{\mathrm{KL}}(p\|q)$, lowering it is exactly lowering the gap between distributions
• With a one-hot target it collapses to $-\log q_y$ — the one line you write in practice
• Paired with softmax the gradient is $q - p$, which never saturates. That's why it beats MSE

For the wider information-theory picture, see Information Theory for AI.