Bayes' Theorem from Scratch — Why a 99% Accurate Test Can Be a Coin Flip

Bayes' theorem is four symbols that reverse a conditional probability, and the single most common way people get probability wrong. This article derives it line by line from the definition of conditional probability, names its four parts, then works a medical test all the way to the number: a test with 99% sensitivity and 99% specificity, applied to a disease 1% of people have, turns a positive result into a 50% chance of being ill. Ends on the base-rate fallacy, sequential updating, and where the prior comes from.

An Essay towards solving a Problem in the Doctrine of Chances

The question you have is the reverse of the one you can measure

A lab can tell you, by running the test on a thousand people who are known to be sick, how often the test says "positive." That is a measurable, reportable number: the test's sensitivity. What a patient wants to know is something else entirely. She has a positive result in her hand, and she wants the probability that she is sick.

These two are not the same number, and confusing them is probably the single most consequential arithmetic error in everyday reasoning. The first is "probability of the evidence, given the hypothesis." The second is "probability of the hypothesis, given the evidence." Bayes' theorem is the machine that turns the first into the second, and it needs one extra ingredient to do it — the one people leave out.

This article builds the theorem from the definition of conditional probability, takes the formula apart symbol by symbol, names its four pieces, and then does the medical-test calculation all the way to the end, because the number at the end is genuinely surprising the first time.

Prerequisite: conditional probability

Write $P(A)$ for the probability that event $A$ happens. Conditional probability, written $P(A \mid B)$ and read "the probability of $A$ given $B$," is the probability that $A$ happens once you already know $B$ happened. Its definition is

$$P(A \mid B) = \frac{P(A \cap B)}{P(B)},$$

where $A \cap B$ means "both $A$ and $B$." The picture: learning that $B$ happened throws away every outcome in which $B$ did not happen, so your world shrinks from everything to just $B$. Inside that smaller world, the part where $A$ also holds is $A \cap B$. So the fraction is "how much of the surviving world is $A$." Dividing by $P(B)$ is what rescales the shrunken world back to a total probability of 1. In words: a conditional probability is a proportion measured inside the world you are left with once you know the condition.

Multiply both sides by $P(B)$ and you get the product rule, which is the same statement with nothing divided:

$$P(A \cap B) = P(A \mid B)\, P(B).$$

In words: the chance that both $A$ and $B$ happen is the chance that $B$ happens, times the share of $B$ in which $A$ also holds.

Now notice that "both $A$ and $B$" does not care which event you name first: $A \cap B$ and $B \cap A$ are the same event. So you can apply the product rule the other way around too:

$$P(A \cap B) = P(B \mid A)\, P(A).$$

Two expressions for the same quantity. That is the entire derivation.

The theorem, one symbol at a time

Set the two right-hand sides equal, since both equal $P(A \cap B)$:

$$P(A \mid B)\, P(B) = P(B \mid A)\, P(A)$$

and divide both sides by $P(B)$:

That is Bayes' theorem. Read it left to right as a sentence. On the left is what you want: the probability of $A$ now that you have seen $B$. On the right, $P(B \mid A)$ is the reverse conditional — the one the lab can measure. $P(A)$ is how likely $A$ was before you saw anything. And $P(B)$ on the bottom is how likely the evidence $B$ was overall, whether or not $A$ held. In words: take the probability in the direction you can measure, multiply by how likely the hypothesis was to begin with, divide by how likely the evidence was at all, and you get the probability in the direction you care about.

Nothing was assumed and nothing was approximated: the theorem is two lines of algebra on the definition of conditional probability. Its power is entirely in what it lets you do — swap the direction of a conditional, as long as you are willing to supply $P(A)$.

That last clause is where the difficulty lives. $P(B \mid A)$ alone does not determine $P(A \mid B)$; you also need to know how common $A$ was to begin with. A test result is evidence, not a verdict, and how much a piece of evidence moves you depends on where you started.

The four words

The pieces have names, and it is worth learning them because every paper and library uses them.

Prior, $P(A)$: what you believed about $A$ before this evidence arrived. In the medical case, the fraction of people who have the disease — the prevalence, or base rate.

Likelihood, $P(B \mid A)$: how probable the evidence would be if $A$ were true. Note it is a statement about the evidence, not about $A$. This is the number experiments report.

Marginal likelihood (or evidence), $P(B)$: how probable the evidence is overall. You almost never get this handed to you; you compute it by adding up every way $B$ could have happened. With two hypotheses, $A$ and not-$A$:

This is the law of total probability, and it is the step people skip. In words, it says: $B$ can happen because $A$ is true and the test fired correctly, or because $A$ is false and the test fired anyway. Both routes count.

Posterior, $P(A \mid B)$: what you believe about $A$ after folding in the evidence. This is the output, and — crucially — it is a probability of the same kind as the prior, which is what makes the whole thing repeatable.

A medical test, worked to the end

Here are the numbers. A disease affects 1% of the population. A test for it has sensitivity 99% — of people who have the disease, 99% test positive — and specificity 99% — of people who do not have the disease, 99% test negative. You test positive. What is the probability you have the disease?

Let $D$ mean "has the disease" and $+$ mean "tests positive." Translate each given number into notation, carefully:

• $P(D) = 0.01$ — the prior, the prevalence
• $P(+ \mid D) = 0.99$ — the sensitivity, our likelihood
• $P(- \mid \lnot D) = 0.99$ — the specificity, so the false-positive rate is $P(+ \mid \lnot D) = 1 - 0.99 = 0.01$
• $P(\lnot D) = 1 - 0.01 = 0.99$ — most people are healthy

Now the denominator, by the law of total probability. There are exactly two ways to get a positive result:

$$P(+) = \underbrace{P(+ \mid D)P(D)}_{\text{sick, correct}} + \underbrace{P(+ \mid \lnot D)P(\lnot D)}_{\text{healthy, false alarm}}$$

$$P(+) = (0.99)(0.01) + (0.01)(0.99) = 0.0099 + 0.0099 = 0.0198$$

Look at those two terms before going on. They are equal. The sick group is small (1%) but almost always tests positive; the healthy group is large (99%) and only rarely tests positive — and those two effects cancel exactly. Half of all positives come from each group. The answer is already visible.

$$P(D \mid +) = \frac{P(+ \mid D)P(D)}{P(+)} = \frac{0.0099}{0.0198} = 0.5$$

Exactly 50%. A test that is right 99% of the time, returning a positive result, leaves you at a coin flip. Not 99%, not 98% — 50%.

The counting version makes it concrete. Take 10,000 people:

• 100 have the disease (1%). Of those, 99 test positive.
• 9,900 do not. Of those, 1% — that is 99 people — test positive anyway.
• Total positives: $99 + 99 = 198$. Of those 198, exactly 99 are actually sick.

$99/198 = 1/2$. No formula required; the theorem is just bookkeeping on this table.

The base rate fallacy: why the answer feels wrong

Almost everyone's first guess is "about 99%." The error has a name — the base rate fallacy — and it is the habit of reading the likelihood $P(+ \mid D)$ as if it were the posterior $P(D \mid +)$. They are different questions, and the theorem shows exactly what bridges them: the base rate $P(D)$.

The mechanism is a competition between two populations. The test's error rate, 1%, is applied to the healthy group, and that group is 99 times larger than the sick group. So the small error rate acting on the huge population produces just as many positives as the high success rate acting on the tiny one. Rarity manufactures false positives, and it does so in proportion to how rare the condition is.

Watch the posterior move as the prior moves, everything else fixed:

Same test, same 99%/99% accuracy, answers from 9% to 99%. The test did not change; only the question's context did. This is why screening an entire population for a rare disease produces a flood of false alarms, while the same test given to someone who already has symptoms — where the relevant prior is not 1% but perhaps 30% — is genuinely informative. It is also why "the test is 99% accurate" is not, on its own, a meaningful statement about your result.

The cleanest way to hold this in your head is the odds form of the theorem. Divide the Bayes expression for $D$ by the one for $\lnot D$ and the awkward $P(+)$ cancels:

In our case the prior odds are $0.01/0.99 = 1{:}99$ and the likelihood ratio is $0.99/0.01 = 99$. Multiply: $99 \times (1/99) = 1$, that is posterior odds of $1{:}1$ — a 50% probability, in one line. The reading is clean — in words: evidence multiplies odds. A test with a likelihood ratio of 99 moves you 99-fold, and 99-fold from 1-in-99 lands you at even money. If you want one takeaway, take this: strength of evidence is a multiplier, so the answer always depends on where you started.

Sequential updating: today's posterior is tomorrow's prior

Because the posterior is a probability of the same kind as the prior, you can feed it straight back in. Suppose you take a second, independent test and it is also positive. Your prior is no longer 1% — it is the 50% you just computed.

$$P(+) = (0.99)(0.5) + (0.01)(0.5) = 0.495 + 0.005 = 0.5$$

$$P(D \mid +,+) = \frac{(0.99)(0.5)}{0.5} = 0.99$$

In words: the second update is the very same formula as the first; only the starting point — the prior — has changed, from 1% to 50%.

Two positives take you to 99%. In odds form it is immediate: posterior odds $= 99 \times 99 \times (1/99) = 99$, i.e. $99{:}1$. Each independent piece of evidence multiplies the odds by its likelihood ratio, so evidence compounds — and note that the order does not matter, since multiplication commutes. This is why doctors confirm a screening result with a second, different test rather than trusting the first, and it is the same mechanism behind every online learning system that revises a belief as data arrives.

One caution buried in that word independent: if the two tests fail for the same reason — the same interfering substance in the blood, the same mislabelled sample — then a second positive carries far less than a factor of 99, and multiplying the ratios overstates your confidence badly.

Pitfalls

You cannot dodge the prior. The commonest complaint about Bayes is that the prior is subjective, and the commonest mistake is to respond by leaving it out — which silently assumes a prior of 50/50. That is not neutrality; on a disease with 1% prevalence, assuming 50/50 is a specific and very wrong claim, and it is what produces the 99% answer people guess. When no base rate is available, the honest move is to state the prior you used and show how the conclusion changes across a plausible range, exactly as the table above does.

Sensitivity is not specificity, and neither is "accuracy." A test's two error rates are independent dials, and a single "99% accurate" figure hides which one it means. Worse, on a rare disease a test that simply always answers "negative" achieves 99% overall accuracy while having zero sensitivity. Always ask for the two numbers separately.

A negative result is also a Bayesian update — usually a very strong one. With the same test, $P(D \mid -) = (0.01 \times 0.01)/((0.01)(0.01) + (0.99)(0.99)) = 0.0001/0.9802 = 0.000102$, about 1 in 9,800. The asymmetry is worth sitting with: on a rare disease the same test is nearly conclusive when negative and nearly useless when positive. Rarity helps you rule out and hurts you when ruling in.

Bayesian and frequentist answer different questions. A frequentist p-value is $P(\text{data at least this extreme} \mid \text{null hypothesis true})$ — a likelihood-flavoured quantity. It is not the probability that the null is true, and converting one into the other requires a prior, which the p-value does not contain. Reading "$p = 0.01$" as "99% chance the effect is real" is the base rate fallacy in its academic costume. The details are in Hypothesis Testing and A/B Tests.

With continuous parameters, the denominator is an integral. For a parameter $\theta$ rather than a two-way hypothesis, $P(B) = \int P(B \mid \theta) P(\theta)\, d\theta$, and that integral is usually intractable. Every heavy machine in Bayesian computation — MCMC, variational inference — exists to work around this one term. See Monte Carlo Methods.

How it is used in practice

Few people at work call it Bayes' theorem by name, but the calculation runs everywhere, every day. Four representative places:

1. Designing alerts for anomaly or fraud detection (data scientists, SREs, fraud teams). Point a detector with "99% detection, 1% false alarms" at transactions where only 1 in 10,000 is fraudulent, and most of its alerts will be false. It is exactly the medical-test calculation: the false-positive term in the denominator is multiplied by a huge normal population. When you evaluate a detector, look beyond recall (= sensitivity) and estimate precision at the production base rate. Precision is $P(\text{fraud} \mid \text{alert})$ — the posterior itself — and it moves whenever the production base rate moves. If humans review alerts one by one, this number sets their workload.

2. Working with a classifier's output probabilities (ML engineers). Train on imbalanced data rebalanced by oversampling or class weights (scikit-learn's class_weight="balanced", for instance) and the prior the model learns no longer matches the production base rate. Feed its predict_proba into thresholds or decisions as if it were a production probability and you will systematically overestimate positives. A prior mismatch can be corrected exactly as the odds form suggests: multiply the posterior odds by "production prior odds ÷ training prior odds." If you need the probabilities themselves, calibrate them (CalibratedClassifierCV and the like) on validation data drawn from a production-like distribution.

3. As a first baseline for text classification (NLP and search teams). Naive Bayes is Bayes' theorem plus one assumption — features are conditionally independent — and it is still used as the baseline for spam filtering and ticket routing. In scikit-learn's MultinomialNB, the knobs you actually touch are alpha, the smoothing that stops an unseen word from zeroing out the likelihood; fit_prior, whether to estimate the prior from the training data; and class_prior, to supply the prior directly. Conditional independence is the same assumption as in the sequential-updating section, and correlated features (several phrasings of the same idea, say) make it overconfident. That is why its scores are fine for ranking but its probabilities should not be taken at face value.

4. Reporting A/B test results (analysts, product teams). "p = 0.01, so there is a 99% chance the effect is real" is precisely the confusion from the pitfalls section. If you want to state the probability that there is an effect, you need a Bayesian analysis with an explicit prior; otherwise report the p-value as "how unusual this data would be if there were no effect."

The pitfall that causes incidents. When the base rate differs between training/evaluation and production — fraud rates that shift with the season, a narrowed target audience — precision can swing hard even though neither the model nor the detector changed. The likelihoods (sensitivity, false-alarm rate) are properties of the detector; the posterior is a property of the detector and the population. When alert quality suddenly drops, first check whether the population's base rate has moved.

Questions you get in interviews and design reviews. The classic: "A detector with 99% sensitivity and 99% specificity runs on data with a 0.1% positive rate. What is its precision?" A good answer builds the per-10,000 count table (about 10 true positives, about 100 false positives), concludes precision is roughly 9%, and says that you therefore need a second-stage check or a revised threshold. Another: "Can you use the probabilities from a model trained on rebalanced data as-is?" Answer: no — the prior is shifted, so correct or calibrate first.

Summary

• Conditional probability $P(A \mid B) = P(A \cap B)/P(B)$ shrinks your world to $B$ and renormalises; applying the product rule both ways gives the theorem in two lines
• $P(A \mid B) = P(B \mid A)P(A)/P(B)$ reverses a conditional — prior, likelihood, marginal likelihood, posterior
• The denominator needs the law of total probability, $P(B) = P(B \mid A)P(A) + P(B \mid \lnot A)P(\lnot A)$, and it is the step most often skipped
• A 99%-sensitive, 99%-specific test for a 1%-prevalent disease gives $P(D \mid +) = 0.0099/0.0198 = 50\%$, because the false positives from the large healthy group exactly match the true positives from the small sick group
• The base rate fallacy is mistaking $P(+ \mid D)$ for $P(D \mid +)$; the odds form, posterior odds = likelihood ratio × prior odds, makes the dependence on the base rate impossible to lose
• Posteriors become priors, so independent evidence multiplies odds: a second positive test takes 50% to 99%

For a fuller tour of priors and posteriors in practice, see Thinking Bayesian; for where this shows up inside models, Bayes' Theorem in AI; and for the probability groundwork underneath, Probability and Statistics for AI.

Exercises

1. Same test (sensitivity 99%, specificity 99%), but the disease affects 1 in 1,000 people. Compute $P(D \mid +)$ exactly, and then explain the result in one sentence using the odds form.
2. A manufacturer improves the test so that specificity rises to 99.9% while sensitivity stays at 99%. With prevalence back at 1%, what is $P(D \mid +)$? Which of the two error rates was the bottleneck, and why?
3. In a clinic, 30% of the people who get tested already have symptoms consistent with the disease, and the test is the original 99%/99% one. A patient from this group tests positive. Compute the posterior. Then explain why the same test can be called "useless" by a population screening programme and "reliable" by this clinic.
4. A colleague argues: "The posterior after two positive tests is 99%, so the third positive would take it to 99.99%. Therefore enough repeats of any test give certainty." Compute the third update, then state the assumption his conclusion depends on and one realistic way it breaks.