Imbalanced Data in Practice — What to Optimize When 99% Is Normal

When only 1% of your data is positive, accuracy lies. We build up from the confusion matrix to PR curves, class weighting, resampling, probability calibration, and deriving the decision threshold from cost — no prior knowledge assumed.

The machine that prints "you're fine" on every slip

Imagine a disease that affects one person in a thousand. You're asked to build a detector, and you build this: an empty box that prints "negative" for everyone who walks in.

Its accuracy is 99.9%. Out of a thousand people, 999 really are negative, so it gets exactly one wrong. On paper it outperforms any sophisticated medical AI. And of course it never finds a single patient.

Fraudulent card transactions, equipment failures, intrusions, customers about to churn, rare diseases — almost everything worth detecting in production is a small slice of the whole. The rarer the thing you're hunting, the less your sense of "it's mostly right" is worth. This article builds up, in order, what to measure in that situation, what to turn, and what you ultimately have to decide.

Start by sorting outcomes into four boxes

Before talking about metrics, fix the counting. A prediction is positive or negative; the truth is positive or negative. That's four combinations, no more.

This is the confusion matrix, and every evaluation metric you'll meet is some arithmetic on these four numbers. The two you need most:

$$\text{Precision} = \frac{TP}{TP+FP}, \qquad \text{Recall} = \frac{TP}{TP+FN}$$

In words: precision is "of the things I raised an alarm about, how many were real", and recall is "of the real things out there, how many did I catch". The numerator is the same in both — the cases you correctly caught. Only the denominator differs. Precision divides by the alarms you raised; recall divides by the number of real cases that exist.

That asymmetry matters on the job. You can always raise precision by raising fewer alarms, because that denominator is yours to control. Recall's denominator isn't. No matter where you move the threshold, the number of true cases in the world stays where it is.

Put the "everyone's fine" machine into the matrix and its trick becomes obvious: TP = 0, FP = 0, FN = 1, TN = 999. Accuracy, $(TP+TN)/N$, is dominated by TN — and TN only counts the things you were right to ignore. Accuracy is a report card for the majority class.

One percent means "you barely have any data"

Here's a second fact that's easy to miss. With 100,000 rows at a 1% positive rate, you have 1,000 positives. So for the phenomenon you actually want to learn, your dataset is not 100,000 examples. It is 1,000.

A model will memorize those 1,000 without breaking a sweat. It nails every minority example in training, then catches nothing new. That is the ordinary failure mode of a flexible model on a small dataset, and imbalance puts you there whether or not your total row count looks large.

The same shortage makes your metrics jittery. If the test set holds 100 positives, one lucky case moves recall by a full point. Declaring "model A wins" on a third-decimal difference is already unsound at that sample size.

ROC vs PR: the difference that bites

Now the single most common misreading in imbalanced problems. An ROC AUC of 0.98 is perfectly compatible with 3% precision in production.

The ROC curve puts recall on the vertical axis and the false positive rate on the horizontal one.

$$\text{FPR} = \frac{FP}{FP+TN}$$

Which says: the denominator is the total number of genuinely negative cases. In a world with 99,000 negatives, emitting 1,000 false alarms moves FPR by 0.01 — visually still glued to the left edge of the plot. Those same 1,000 false alarms land directly in precision's denominator. Catch 100 positives alongside 1,000 false alarms and precision is about 9%: nine out of every ten alerts your analysts open are nothing.

• The ROC baseline (what random guessing scores) is 0.5, whatever the positive rate is
• The PR baseline is the positive rate $\pi$ itself — 0.01 when positives are 1%

So ROC quietly launders the imbalance away, while the PR curve shows it to you at full strength. When the thing you're looking for is rare, make the PR curve your primary metric. The standard summary of the area under it is Average Precision (AP): sweep the threshold and, each time recall increases, add in the precision you had at that moment.