#cross-entropy

2 articles

01 ·Information Theory·FREE·9 min read Entropy and Cross-Entropy — Where the Loss Function Comes From The cross-entropy loss you write in every classifier wasn't handed down by decree. Starting from the idea of surprise, we walk a single path through entropy and cross-entropy to that familiar one-liner — and finish with why squared error is the wrong tool for the job. 02 ·Information Theory·FREE·9 min read Information Theory and AI — Where Cross-Entropy Loss Came From Information is surprise, entropy is average surprise, KL divergence is the gap between two distributions. Stack those three and the loss function you already use for classification falls out as the only reasonable answer — including why it involves a logarithm, and what perplexity is really counting.