Thinking Bayesian — A Working Feel for Priors, Likelihoods, and Posteriors

Bayesian updating is the act of feeding yesterday's posterior back in as today's prior. With a conjugate prior the whole update collapses into adding pseudo-counts, and an A/B test becomes two numbers: the probability of winning and the expected loss. From zero background to the traps in stopping rules and prior choice.

You changed the button color — did it actually help?

You repainted the buy button from blue to green. After a week, the old design was shown 1,000 times and clicked 40 times; the new one was shown 1,000 times and clicked 50 times. That is 4.0% versus 5.0%. Do you get to report "a 25% lift"?

If you flip a coin ten times and see six heads, nobody says the coin is biased. Same here: at a thousand impressions, a gap between 40 and 50 clicks happens by luck all the time. At this point most teams reach for a p-value and call it black or white depending on whether it lands under 0.05.

But in a decision meeting, "we cannot rule out that there is no difference" is not what anyone actually wants to know. The two real questions are: how likely is it that the new button is genuinely better, and if we switch and we are wrong, how much do we lose? Bayesian thinking answers both directly. The introduction to the theorem itself lives in Bayes' theorem in AI; this article picks up where that leaves off, focusing on what repeated updating feels like and how to actually run an A/B test with it.

Intuition: belief is a pile of chips spread across candidates

Call the true click rate of the new button $\theta$. Since you do not know it, do not commit to a single number. Chop the possibilities up — "maybe 1%, maybe 2%, … maybe 10%" — and spread chips across the candidates. That spread is the prior.

Every time data arrives, you multiply each candidate's chips by a number. That number is "if this candidate were the truth, how natural would the data I just saw be?" — the likelihood. If a click was observed, the 10% candidate receives a bigger multiplier than the 1% candidate. Divide everything so the chips sum to one again, and you have the posterior.

Two properties matter later. First, updating is multiplication. It is sculpture — candidates that do not fit get carved away — which means a candidate you set to zero in the prior stays at zero no matter what data arrives. The moment you declare something impossible, it can never come back. Putting a zero in your prior is a much stronger statement than it looks.

Second, updating is stackable. The posterior you built from day one's data can serve as day two's prior, and the result is identical to pouring all seven days in at once. Order does not matter either. That fits a world where data piles up daily.

Mechanics: multiply beliefs, add evidence

Everything above fits into one proportionality.

Here $\theta$ is the unknown you care about (the click rate), $D$ is the data you observed, the bar $\mid$ means "given that", and $\propto$ means "is proportional to". You can ignore constant factors because the final rescale-to-one step absorbs them anyway. Read aloud: the posterior is proportional to the likelihood times the prior. That is the whole thing.

Strip the notation away and one line of English is left, which says that your belief after seeing the data is your belief before it, re-weighted by how well each candidate explains what showed up. It is the chips-and-multipliers routine from the previous section, written in symbols.

With exactly two candidates it gets even friendlier. Take the ratio of both sides and then the logarithm, and multiplication turns into addition.

The left side is the log of "after seeing the data, how many times more plausible is candidate 1 than candidate 2". The first term on the right is "how much better candidate 1 explains that data" — the weight of the evidence — and the second term is what you believed beforehand. So evidence is a score you add on top of your prior stance. Put in words: your leaning after the data equals what the data argues plus what you already leaned, on one shared scale.

Written this way, a common worry resolves itself. The first term keeps accumulating as data arrives; the second term never changes. So with enough data the choice of prior does not change the conclusion, and the prior only bites while data is scarce. "Priors are arbitrary" is a fair complaint exactly in the small-data regime and nowhere else.