Probability and Statistics for AI — A Model's Output Is a Distribution

Classifiers and language models do not return answers; they return probability distributions. Distributions, expectation, conditional probability and Bayes explained from the symbols up — building to the payoff: why maximum likelihood is where loss functions come from. Cross-entropy and MSE were derived, not invented.

The goal: being able to read this equation

When a paper writes down what it is training for, it almost always uses this shape.

Written out in words it says: show the model some data, ask how much probability it put on the correct answer, take the log of that number, average over everything you showed it, and flip the sign. $\theta$ is the model's parameters, $\mathbb{E}$ is an average, $\mathcal{D}$ is where the data comes from, $\sim$ means "drawn from", $p_{\theta}(y \mid x)$ is the probability the model assigns to $y$ having seen $x$, and the vertical bar $\mid$ reads "given".

The notation is unfamiliar; the content is not. And once this line is readable, classification losses and language-model losses turn out to be the same equation wearing different clothes.

What a model returns is not an answer

Nothing else works until this is dislodged. A classifier does not say "cat". It returns a probability for every class it knows — cat 0.82, dog 0.15, other 0.03. The interface shows you the largest entry, which is why it looks like a single answer.

Language models do the same thing. For every next token, they build a bar chart over the entire vocabulary, tens of thousands of entries wide. After "the capital of Japan is", perhaps Tokyo 0.71, Osaka 0.04, and so on down the list.

The function that builds that chart is softmax.

$z_i$ is the raw score the model emits — the logit, which can perfectly well be negative — $\exp$ is the exponential, and the denominator sums over every entry.

The fraction, in words, does two things: make every score positive without disturbing which one is largest, then rescale the whole set so it adds up to one. Calling something a probability requires exactly two properties, non-negative and summing to one; $\exp$ delivers the first, the division delivers the second. There is no deeper philosophy in it.

Expectation: the formal way to write "average"

$$\mathbb{E}[X] = \sum_{x} x\,p(x)$$

$X$ is a quantity whose value is decided by chance and $p(x)$ is the probability of it taking value $x$.

Put in words, the sum is an average in which every possible value is weighted by how likely that value is. If a quantity is 1 with probability 90% and 100 with probability 10%, its expectation is not the naive 50.5 but $1 \times 0.9 + 100 \times 0.1 = 10.9$.

When $\mathbb{E}$ appears in a paper, reading it as "average this over the data" will be right nine times out of ten. Implementations cannot afford the full dataset every step, so they substitute the mini-batch average. That is what the $\mathbb{E}$ in equation (1) becomes in code.

Conditional probability: what a language model is doing

$p(y \mid x)$ is the probability of $y$ given that $x$ has happened. Predicting umbrella sales cold is hard; predicting them given "it is raining" is a different exercise. Conditional probability is the notation for "new information changes the outlook".

A language model is that idea wired in series.

$\prod$ ("pi") means "multiply all of these", $x_t$ is the $t$-th token, and $x_{<t}$ means every token before it.

It is one line which says: the probability of a whole passage is the product of, at each position, the probability of that token given everything written so far. A GPT-style model computes only the individual factors on the right; generation is drawing from them one at a time. This is what "it just predicts the next word" actually refers to.