KPI & Statistical Decision Making

Where this is used

Did version A beat version B, and is that gain real or just noise? A/B testing and p-values are how you answer that with numbers instead of gut feel. Once you can design a KPI, read a confidence interval and work out how big a sample you need, you can back up what you say in a meeting, put ad money where it actually pays off, and tell a genuine quality improvement from a lucky week. In a world where decisions are made on data, this is table stakes.

Start gently

A KPI is a number you use to see the state of the work. Never read one figure on its own — read the change and the distribution together.

Key points

What Makes a Good KPI (the SMART Framework)

A KPI (key performance indicator) should be Specific, Measurable, Achievable, Relevant and Time-bound. "Grow revenue" is a wish, not a KPI. "Increase new customers by 20% over last quarter by the end of this quarter" is a KPI. The other thing to get right is the mix of leading KPIs, which track the activity you do now, and lagging KPIs, which track the results that follow.

The t-Statistic in an A/B Test

This is the number that tells you whether the gap between the average for A and the average for B is random wobble or a real difference. The larger |t| is, the less likely the gap is chance. The larger n is, the smaller a difference you can still detect — that is what statistical power means.

p-Values and Significance Levels (Evidence That It Wasn't Luck)

A p-value is the probability of seeing a difference at least as big as the one you got, assuming the true effect is zero. By convention, p < 0.05 (5%) is called statistically significant. A smaller p-value is stronger evidence that chance alone does not explain the result — but p < 0.05 does not mean "definitely true," only "unlikely to be a fluke." In business you should always read it alongside the size of the effect, not on its own.

Confidence Intervals (Putting a Number on Your Uncertainty)

When you estimate a population value from a sample, a 95% confidence interval is the range your method captures the true value in 95% of the time. A narrower interval means a more precise estimate, and bigger samples give narrower intervals. In practice you report it like this: "Version A lifted conversion by 1.2% ± 0.3% (95% CI)."

Sample Size and Statistical Power

Whether an A/B test can spot a difference at all comes down to sample size. Too small a sample and you miss a real effect — that is a Type II error. You work out the sample size you need from three things: the smallest effect worth detecting, the significance level, and the power you want (usually 80%). So "we ran it for a week and saw no significant difference" may not mean there is no effect; it may just mean you never had enough data to see it.

See it drawn

Monthly retention rate (a KPI over time)

Month to month the change looks small, but over six months it is down 11 points — a decline no single-month comparison would reveal.

Jobs that use this

Growth Manager$140k

Quality Assurance Engineer (QAE)$100k

Marketing Analyst$95k

KPI & Statistical Decision Making

Start gently

Key points

See it drawn

Jobs that use this

Members-only from here

Comments