#mutual-information

1 articles

01 ·Information Theory·★ MEMBER·PAPER·11 min read Mutual Information — Putting a Number on What You Know How much does knowing one thing shrink your uncertainty about another? Mutual information turns that into a single number. We build it from entropy subtraction, work through conditional MI, the data processing inequality and why estimation is hard, and end at why the InfoNCE loss behind modern contrastive learning is called a lower bound on mutual information.