Information Theory
Entropy, KL divergence, cross-entropy — where loss functions come from
01
·Information Theory·★ MEMBER·9 min read
Information Theory and AI — Where Cross-Entropy Loss Came From
Information is surprise, entropy is average surprise, KL divergence is the gap between two distributions. Stack those three and the loss function you already use for classification falls out as the only reasonable answer — including why it involves a logarithm, and what perplexity is really counting.
02
·Information Theory·★ MEMBER·PAPER·9 min read
KL Divergence From Scratch — Measuring the Gap Between Two Distributions
KL divergence measures the gap between two probability distributions. We build it up from a compression metaphor to the definition, its famous asymmetry, and a numpy implementation — then watch it at work as the regularizer in VAEs and the leash in RLHF.
03
·Information Theory·FREE·9 min read
Entropy and Cross-Entropy — Where the Loss Function Comes From
The cross-entropy loss you write in every classifier wasn't handed down by decree. Starting from the idea of surprise, we walk a single path through entropy and cross-entropy to that familiar one-liner — and finish with why squared error is the wrong tool for the job.
04
·Information Theory·★ MEMBER·PAPER·11 min read
Mutual Information — Putting a Number on What You Know
How much does knowing one thing shrink your uncertainty about another? Mutual information turns that into a single number. We build it from entropy subtraction, work through conditional MI, the data processing inequality and why estimation is hard, and end at why the InfoNCE loss behind modern contrastive learning is called a lower bound on mutual information.
05
·Information Theory·★ MEMBER·PAPER·11 min read
Compression Is Prediction Is Intelligence — LLMs Through Information Theory
Training a model to guess the next token is training it to shrink a file. Route the prediction through arithmetic coding and "probability model" and "code" turn out to be the same object — which makes cross-entropy loss literally the size of the compressed output. Built up from zero, ending at why the Hutter Prize treats compression ratio as an intelligence test.