#perplexity

1 articles

01 ·Information Theory·FREE·9 min read Information Theory and AI — Where Cross-Entropy Loss Came From Information is surprise, entropy is average surprise, KL divergence is the gap between two distributions. Stack those three and the loss function you already use for classification falls out as the only reasonable answer — including why it involves a logarithm, and what perplexity is really counting.