Hypothesis Testing and A/B Tests — How to Use a p-value, and How People Misuse It

Ship the identical page to two groups and the numbers still won't match. A hypothesis test converts that gap into 'how many units of noise is this?' — that number is the p-value. From the null hypothesis up through the three failures that actually ruin experiments: power, peeking, and multiple comparisons.

Your button change lifted clicks. Did it?

You ship two versions of a checkout button and split traffic evenly. A converts at 5.0%, B at 5.4% — a 0.4-point lift. Before you say "let's ship B" in the review meeting, there's one thing to check: two identical pages, given identical traffic, will not produce identical click rates.

Try flipping a fair coin 100 times, twice. Getting exactly 50 heads both times would be the surprising outcome. You get 48 and 53, or 45 and 52. Even when the two things being compared are literally the same, a gap appears. So the existence of a gap proves nothing at all. The question worth asking is not is there a difference but how often would a gap this large show up even if the two versions were identical?

A hypothesis test is the tool that answers that question with a number. The p-value is that answer.

The null hypothesis is a presumption of innocence

A test works by contradiction. You start by assuming the opposite of what you want to believe.

• Null hypothesis $H_0$: A and B have the same true click rate (the difference is zero)
• Alternative hypothesis $H_1$: they don't

Then you compute how unusual today's data would be if $H_0$ were true. If it's unusual enough, you conclude the assumption was the problem and throw $H_0$ out.

Courtrooms work the same way. The defendant is presumed innocent, and is convicted only when the evidence would be absurd under that presumption. And an acquittal is not proof of innocence — it means the evidence wasn't strong enough to convict. Testing inherits this exactly: failing to reject $H_0$ is not proof that there is no difference. Confusing those two is the single most common mistake in practice.

Here is the definition:

p-value = the probability of observing a difference at least as extreme as the one you saw, purely by chance, assuming the null hypothesis is true.

Watch the direction of the conditioning. A p-value is "how surprising the data is, given $H_0$" — not "how likely $H_0$ is, given the data." p = 0.03 does not mean there's a 97% chance B is better. Getting from the first to the second requires a prior, which is exactly what Bayesian thinking does head-on.

Measuring how much noise wiggles

To judge "unusual," you first need a ruler for "how much does this wiggle by chance alone?" For a click rate $\hat p$ estimated from $n$ visitors, that ruler is the standard error:

$$\mathrm{SE}(\hat p)=\sqrt{\frac{p(1-p)}{n}}$$

Here $p$ is the true click rate and $n$ is how many people you measured. Put in words: a rate measured on n people jitters by roughly this much every time you measure it. The width shrinks with the square root of $n$, which is a brutal exchange rate — halving your error bar costs four times the traffic. (That same $1/\sqrt{n}$ crawl is why Monte Carlo methods converge so slowly; it's the same root cause.)

Express the observed gap in units of that ruler and you have the test statistic:

The numerator is the gap you actually saw. The denominator is how large that gap typically gets by chance when $H_0$ holds. $\bar p$ is the click rate pooled across both groups — under $H_0$ they share one true rate, so you estimate it jointly. Equation (1) is therefore doing one thing only: counting how many units of noise your gap is worth. Once $|z|$ passes roughly 1.96, the two-sided p-value drops below 0.05.

Why 1.96? Under $H_0$, $z$ approximately follows the standard normal distribution — the bell curve with mean 0 and standard deviation 1 — and 95% of that curve sits between $\pm 1.96$. The familiar "two-sided 5%" is just the line that trims 2.5% off each tail. And there is nothing natural about 5%: it's an early-20th-century convention that being fooled one time in twenty is tolerable. The threshold is yours to move to fit the stakes. What you may never move is the threshold after you've looked at the data.

From here on it's not about usage — it's about misuse

That's a textbook chapter's worth of material. But what actually breaks experiments in the field is not arithmetic on that formula. It's looking at the data first, and only then choosing the convenient slice, the convenient stopping point, the convenient metric. This isn't really a statistics-specific disease. It's the shape every ML practitioner already knows: the harder you fit what's in front of you, the worse you do in production.