Mutual Information — Putting a Number on What You Know

How much does knowing one thing shrink your uncertainty about another? Mutual information turns that into a single number. We build it from entropy subtraction, work through conditional MI, the data processing inequality and why estimation is hard, and end at why the InfoNCE loss behind modern contrastive learning is called a lower bound on mutual information.

Representation Learning with Contrastive Predictive Coding

Putting a number on "how much does knowing help?"

You're standing at the door wondering whether to take an umbrella. Someone says: "By the way, the pressure dropped sharply this morning." How much clearer did that one sentence make your picture of whether it's going to rain?

Mutual information (MI) is exactly that "how much clearer" turned into a number. Knowing one variable shrinks your uncertainty about another by some amount; the bigger the shrinkage, the stronger the relationship. That's the whole idea.

Correlation plays a similar role, but it only measures whether two things rise together in a straight line. Take $y = x^2$ with $x$ wandering between $-1$ and $1$: the correlation is essentially zero, because $y$ grows the same way whether $x$ is positive or negative, and no straight line fits that. Yet knowing $x$ pins down $y$ completely. Mutual information correctly reports that the relationship is as strong as it gets. It picks up dependence of any shape, and that is its main selling point.

This article sits next to Information Theory and AI, but assumes no prior background.

An analogy: what one question is worth in Twenty Questions

Someone thinks of a word and you have to guess it with yes/no questions. If there are 1024 candidates, your uncertainty is worth ten questions (since $2^{10} = 1024$). That "amount of uncertainty" is entropy, measured in bits.

Now you ask "is it a living thing?" and the field narrows to 256 words. Your uncertainty went from 10 bits to 8. That question was worth 2 bits.

Mutual information is precisely that worth — except it isn't one lucky question, it's the average over every way the answer could come back, weighted by how likely each one is. A question that only narrows things down when the answer is "yes" is worth almost nothing on average if "yes" is rare. A question that splits the field in half either way earns a clean bit every time. A good question is one whose answer you can't predict — that's what this average is saying.

The mechanism: MI is a subtraction of uncertainties

Put plainly, mutual information is a subtraction. Take your uncertainty about $Y$ when you know nothing, and subtract the uncertainty about $Y$ that remains after you're told $X$. Being a difference of that kind, it's always at least zero — being told something never makes you more uncertain on average. And if the difference is zero, $X$ says nothing whatsoever about $Y$. That is independence, stated in information-theoretic terms.

Let's build it up. First, uncertainty itself — entropy. Here $p(x)$ is the probability that $X$ takes the value $x$.

In words: take how surprising each outcome is, $-\log p(x)$, and average it using how often that outcome actually happens. If the same few outcomes keep showing up, entropy is small; if you have no idea what's coming, it's large. Use base-2 logs and the unit is bits; use natural logs and it's nats.

Next, the uncertainty left after learning $X$ — conditional entropy.

That is: "how uncertain am I about $Y$ in the world where $X$ turned out to be $x$," averaged over how likely each $x$ is. The more $X$ narrows down $Y$, the smaller this gets.

Read the vertical bar as "given that you've been told": in words, $H(Y \mid X)$ is the uncertainty about $Y$ that survives even after someone hands you $X$.

Mutual information is the gap between the two.

Spelled out in words: start with the uncertainty you walked in with, subtract whatever is still left after the tip-off, and the leftover difference is $I(X;Y)$.

Two equals signs in a row is a real claim: you get the same number from either direction. What $X$ tells you about $Y$ always equals what $Y$ tells you about $X$. It's counterintuitive but follows straight from the definition, and it's why the quantity is called mutual.

A peaked distribution means low uncertainty, a flat one means high. Getting that correspondence into your hands makes everything below easier.

A second face: distance from independence

So far we defined MI as a subtraction. The very same quantity can be written from a different angle.

Here $p(x,y)$ is how the two actually show up together, and $p(x)p(y)$ is a fictional distribution describing how they would show up if they were unrelated. $D_{\mathrm{KL}}$ measures the gap between two distributions (KL Divergence from Scratch), so this equation reads: "how far is the real pairing from the pairing you'd get under independence?"