A Field Guide to Probability Distributions — Where Normal, Poisson, and the Exponential Family Come From

The normal and Poisson distributions aren't formulas to memorize — they're the shapes that certain situations inevitably produce. Follow three stories (adding things up, counting rare events, and refusing to assume more than you know) and the textbook zoo collapses into one river called the exponential family, with sigmoid and softmax falling out of it.

When the table of distributions looks like a stack of flashcards

Open any probability textbook and you'll hit the catalogue: Bernoulli, binomial, Poisson, geometric, exponential, gamma, beta, normal, log-normal, Dirichlet. Each comes with a forbidding density function and a pair of formulas for mean and variance. This is where a lot of readers decide there's too much to memorize and close the book.

But the people who use these fluently at work aren't reciting formulas. What they carry around is a story about the situation each distribution comes out of. With the story in hand, one look at the data narrows things down: this is a sum of many small things, so probably normal. The formula can be looked up in the scipy.stats docs afterward.

This article organizes the zoo into three stories.

1. Add enough things together and you get a normal distribution (the central limit theorem)
2. Count something that rarely happens and you get a Poisson (the rare-event limit)
3. Refuse to assume anything you weren't told and you get the exponential family (maximum entropy)

The third story turns out to swallow most of the other two, and out of it drop sigmoid and softmax. If the notation of probability itself is still shaky, Probability and Statistics for AI lays the groundwork first.

Groundwork: a distribution is a way of handing out plausibility

A distribution hands out "how likely" across the possible outcomes. A die gives 1/6 to each of six faces. That's the discrete case, where each value gets a probability directly — a probability mass function.

Continuous values like height work differently. The probability of being exactly 170.000… cm is zero: on an infinitely fine ruler, no single point has any area. So in the continuous case we hand out height instead. That's the probability density function, and what becomes a probability is not the height itself but the area you cut out over an interval. A density is allowed to exceed 1 — a uniform distribution on $[0, 0.5]$ has density 2 everywhere, which times a width of 0.5 gives area 1, no contradiction. Believing "density = probability" is the first trap.

Story 1: add enough of anything and it turns normal

Flip a coin once, heads = 1 and tails = 0, and you have two bars. Flip ten times and count the heads: now there's a hump in the middle. At a hundred flips, at a thousand, the hump keeps smoothing out into a symmetric bell.

The striking part is that what you started with barely matters. Coins, dice, uniform random numbers between 0 and 1 — draw independently many times, add them up, and the way the average scatters converges to the same shape. That's the central limit theorem.

Unpacking the symbols: $X_1, X_2, \dots$ are values drawn independently from the same distribution, $\bar{X}_n$ is the average of $n$ of them, $\mu$ is the true mean, $\sigma$ is the spread (standard deviation), $\mathcal{N}(0,1)$ is the normal distribution with mean 0 and standard deviation 1, and the $d$ over the arrow means "converges in shape as a distribution." Equation (1) says: measure how far the sample average strays from the true mean using $\sigma/\sqrt{n}$ as your ruler, and whatever you started with, you land on a fixed bell curve. Read it once more in plain words: averaging is itself a shape-making operation, and the shape it makes is always the same bell, whatever went into it. Average test scores, average server response times — collect enough of them and the wobble has the same face.

Two practical consequences follow. First, it explains why "assume the errors are normal" is so often reasonable. Measurement error and model residuals are usually the sum of countless small causes, and sums drift toward normal. Second, look at that $\sqrt{n}$. Ten times the precision on an average costs a hundred times the samples. A/B tests running longer than expected, and evaluation metrics that stay noisy no matter how much you grow the eval set, are both this square root at work.

You can watch it happen. The ingredient is a flat, boxy uniform distribution — nothing bell-shaped about it — yet the averages track theory exactly.

import numpy as np
rng = np.random.default_rng(0)

for n in [1, 2, 10, 100]:
    m = rng.random((200_000, n)).mean(axis=1)   # average n uniform draws
    print(n, round(m.std(), 4), round(np.sqrt(1 / 12 / n), 4))
    # left: measured spread. right: the σ/√n prediction (uniform has variance 1/12)

Remember one precondition: the variance must be finite. Break that and the theorem stops helping. For heavy-tailed quantities — price moves, network latency, file sizes — averaging doesn't pull you toward normal; the occasional enormous value drags the average around instead. And when causes stack up multiplicatively rather than additively (growth rates, prices), taking logs turns the product into a sum, which is why those quantities land on the log-normal.

Story 2: count something rare and you get a Poisson

How many requests arrive in a minute. How many defects remain in a thousand lines of code. These are counts where the number of opportunities is enormous and the per-opportunity probability is tiny.

Take the binomial, send $n \to \infty$ and $p \to 0$ while holding the product $np = \lambda$ fixed, and what survives in the limit is the Poisson distribution.

Here $k$ is how many times it happened and $\lambda$ is how many times it happens on average. Equation (2) returns the probability that a phenomenon averaging $\lambda$ occurrences happens exactly $k$ times — which says, concretely, "on a site that normally takes three requests a minute, how surprising is a minute that takes seven?" The $\lambda^{k}$ on top carries "the more occurrences, the rarer," and the $k!$ underneath divides out the orderings, so the same $k$ events aren't counted once per shuffle. Its defining quirk: the mean and the variance are both $\lambda$. There is only one knob, and it silently insists that a larger average must come with proportionally larger scatter.

import numpy as np
from scipy.stats import binom, poisson

n, p = 10_000, 3e-4          # np = 3
k = np.arange(0, 12)
print(np.abs(binom.pmf(k, n, p) - poisson.pmf(k, n * p)).max())
# with large n and small p, binomial and Poisson barely differ

Look at the same phenomenon from the other side — not "how many," but "how long until the next one" — and the exponential distribution appears. It has a strange property.

The vertical bar means "given that." Equation (3) says that having already waited $s$ changes the odds of waiting another $t$ not one bit. This is called memorylessness — which says, in plain words, that "it's about due" does not exist in this distribution. Glance at the clock partway through and everything ahead of you has rewound to exactly the outlook you had when you started waiting. Wait thirty minutes at the bus stop and your expected remaining wait is exactly what it was when you arrived — counterintuitive, but that's the world the exponential describes. It is the only continuous distribution with this property; on the discrete side, the geometric is the only one.

The wait until the $k$-th arrival is a gamma distribution, and so on — the whole family is related by these transformations.

Story 3: don't decide what you weren't told

The third story is the most powerful. The maximum entropy principle, put on a firm footing by the physicist E. T. Jaynes in 1957, says this:

Many distributions satisfy the constraints you know. Among them, pick the one with the largest entropy.

Entropy measures how spread out a distribution is — equivalently, how little it commits to (see Entropy and Cross-Entropy). Choosing maximum entropy means adding no assumption beyond the constraints you actually have. It's the humblest available answer.

What's surprising is how much of the textbook falls out of that single policy.

So we now have two independent reasons why the normal shows up everywhere. Story 1: it emerges as the limit of sums. Story 3: it's the least presumptuous choice when mean and variance are all you know. One is a fact about the world; the other is a confession of our ignorance.

That last row connects straight to generative models. When picking the next token, the model holds only scores (logits) and wants to claim nothing further. The maximum-entropy answer under exactly that constraint is softmax. The temperature parameter is the inverse of how hard the constraint binds: lower it and the distribution commits harder to the top scores (lower entropy); raise it and everything flattens toward uniform.

The confluence: one river called the exponential family

Nearly every distribution so far can be rewritten in a single form.

Term by term: $x$ is the observed value; $\eta$ (eta) is the natural parameter, the knob on the model's side; $T(x)$ is the sufficient statistic, the summary extracted from the data; $A(\eta)$ is the log-partition function, the correction that makes everything integrate to 1; and $h(x)$ is a base measure that doesn't depend on $\eta$. In plain words, equation (4) says: multiply the data's summary $T(x)$ by the knob $\eta$, exponentiate, and normalize. The whole content of the model is that one multiplication between a knob and a summary; the exponential and $A(\eta)$ are packaging that turns the product into something probability-shaped — never negative, adding up to one. Distributions that fit this mold form the exponential family.

It looks abstract until you substitute something. Rewrite the Bernoulli distribution (probability $p$ of heads) and you get $\eta = \log\frac{p}{1-p}$ — the logit — with $T(x) = x$ and $A(\eta) = \log(1 + e^{\eta})$. Now differentiate $A$ with respect to $\eta$.

That line is short but it earns a reading in plain words: nudge the bookkeeping term $A$ — the one that was only there to make the area come out to 1 — a little way along the knob $\eta$, and the size of its response is the probability of heads. Sigmoid is the derivative of the log-partition function. Do the same for the categorical distribution and you get $A(\eta) = \log\sum_j e^{\eta_j}$, whose gradient is exactly softmax. Putting a sigmoid or a softmax on a network's output isn't a matter of someone picking a convenient squashing function — it's that "take the mean of an exponential family whose natural parameter is the logits" has that shape. Once this clicks, using cross-entropy as the classification loss stops feeling handed down from above and starts feeling forced.

Three reasons the exponential family earns its keep in machine learning:

• Data compresses. The likelihood depends on the data only through $T(x)$. A million records reduce to a sum of sufficient statistics.
• Maximum likelihood becomes convex. The log-likelihood is concave in $\eta$, so there are no local optima to get stuck in.
• Conjugate priors exist. Beta ⇄ Bernoulli, gamma ⇄ Poisson, Dirichlet ⇄ categorical: the posterior closes back into the same shape as the prior. That's why updating on a new observation can amount to incrementing a count.

There are exceptions, of course. The heavy-tailed Student's t, mixtures of Gaussians, and the uniform distribution with unknown endpoints all sit outside the family. Knowing they're outside is useful — it explains why the math suddenly gets harder when they show up.

The field guide: where each one comes from, and where it shows up in AI

None of this needs memorizing. Read the middle column, ask which one your data was generated by, and the candidates drop to two or three.

How this gets used in practice

Who reaches for it, and when. ML engineers choosing a loss function. Data scientists sizing an A/B test. Demand-forecasting and ad-delivery teams regressing on counts. SREs setting a latency SLO. Every one of these is a moment where you assume a shape for a distribution — and a wrong assumption fails quietly rather than loudly.

Names you'll actually type. In PyTorch, torch.distributions (Normal, Poisson, Categorical, Dirichlet) and the matching nn.GaussianNLLLoss / nn.PoissonNLLLoss. For count regression, statsmodels GLM with family=sm.families.Poisson() or NegativeBinomial(). For fitting and quantiles, scipy.stats (.fit(), .ppf()). Weight initialization via nn.init.kaiming_normal_ draws from a normal with standard deviation $\sqrt{2/\text{fan\_in}}$, chosen so the variance of activations holds steady across layers — another place a normal assumption is doing structural work. And the temperature you tune at generation time is precisely the maximum-entropy knob from earlier.

Traps that turn into incidents.

• Using the mean on a heavy-tailed quantity. Latency and cost are nowhere near normal, and the mean gets dragged by a handful of huge values. p50/p95/p99 is the working standard; "average latency looked fine" is usually a statement with no content.
• Fitting count data with MSE. Squared error on counts produces negative predictions and over-weights the high-$\lambda$ region. Poisson loss with a log link is the default alternative.
• Missing overdispersion. Poisson forces variance to equal the mean. If real variance clearly exceeds it — bursts, day-of-week effects, per-user heterogeneity — your prediction intervals come out too narrow and the alerts never stop. That's the moment to try a negative binomial.
• Trusting memorylessness unconditionally. Assume an exponential and you cannot express wear-out, where longer uptime means higher failure risk. If the hazard rate changes over time, look at Weibull and friends.
• Applying the CLT to something with infinite variance. The expectation that more samples will bring normality simply isn't repaid. Get in the habit of taking logs and checking the tail first.

How it gets asked in interviews and design reviews. "Why is classification loss cross-entropy rather than squared error?" — because the output is a categorical distribution, and its maximum-likelihood estimate is cross-entropy. "What loss for a model predicting daily click counts?" — Poisson first; negative binomial if variance exceeds the mean. "Why not state a latency SLO as an average?" — because the distribution is close to log-normal and the mean isn't a representative value. In each case, tracing one story back to its origin gives you the answer.

Summary

• Distributions aren't things to memorize; they sort by the story that generates them. Sums give you the normal, rare counts give you the Poisson family, and declared ignorance gives you maximum entropy
• The CLT's preconditions are independence, identical distribution, and finite variance. Heavy tails void it
• Impose constraints under maximum entropy and out come uniform, exponential, normal, and softmax in turn
• The result is the exponential family, $p(x\mid\eta) = h(x)\exp(\eta^\top T(x) - A(\eta))$, in which sigmoid and softmax are nothing more than the gradient of the log-partition function $A$
• Production incidents in this area almost always trace back to an unexamined assumption about the shape of a distribution

For the next step, KL Divergence covers the tool for measuring the gap between two distributions — which is where the story of distributions turns into the story of optimization.