Chapters 9, 10 and 11 each left something on deposit with the line "that is an evaluation question, for another chapter." This chapter collects all three.

Part III ends with measurement for a reason. Every one of the eleven chapters so far has the shape "change this and that gets better" — and whether you are allowed to say "better" rests entirely on this chapter.

Measuring is not producing one number. It is producing a width and a direction along with it.

1. A single number has a width

A test set is a sample drawn from the whole population of problems you care about. Draw a different 1,000 and the same model's score moves. If the model is right on a fraction $p$ of $N$ items, the number correct follows a binomial distribution and the standard deviation of the fraction is

where $p$ is the true accuracy and $N$ the number of items. The 95% half-width is 1.96 times that, so at $p=0.70$:

A "70%" measured on 100 items is really somewhere between 61% and 79%. It is the same $\sigma/\sqrt{n}$ shape as Part 0 Chapter 3, so halving the width costs four times the items.

Read the other way, the width you want fixes the number of items you need (at $p=0.70$):

So claiming a half-point improvement takes 30,000 items per model. That is one model, measured once.

2. When the width you labelled 95% is not 95%

The $\pm 1.96\sigma$ above is an approximation that holds when the accuracy is away from 0 and 1. How well it holds can be checked without any approximation by enumerating the binomial: fix the true $p$, add up the probability of every $k=0,1,\dots,N$, and read off how often the interval built from the measurement contains the true value.

In the middle the naive formula is fine. The trouble is at the edges: measure a 0.95 model on 100 items and the interval you called 95% actually contains the true value 87.8% of the time. Worse, in the same setting 25.2% of the time the upper end goes above 1 — an interval reaching "103% accuracy" once every four runs.

The closer a test is to saturation, the more this bites. Wilson's formula

with $\hat p$ the measured accuracy, $z=1.96$ and $N$ the item count, pulls the centre slightly inward and never exceeds 1. As the table shows, it is better at the edges and slightly conservative in the middle.

(The table comes from enumeration. Set the knobs in the Section 4 figure to "100 items, true accuracy 0.95" and the measured share — counted by drawing, not enumerating — drops to just under 90% as well.)

3. To compare two models, give them the same problems

Now the main business. Compare models A and B. I built 1,000 artificial items with the following mix (this mix is an assumption I chose):

True accuracy is 0.67 for A and 0.71 for B, and the two agree on 86% of the items. Real tests look like this too: easy items both solve, hard items both miss.

Measured on the 1,000 items I drew:

The two intervals overlap by 0.0147. This is where it is common to write "the intervals overlap, so we cannot claim a difference." Build the interval for the difference itself and the story changes.

Items they agree on say nothing about the difference. Only the items they disagree on carry it: 45 where only A was right, 87 where only B was right, 132 in total (13.2%).

Same data, and the standard error is 1.79 times smaller. Pairing subtracts away the spread in item difficulty.

Looking only at the 132 disagreements, you can test this with almost no assumptions. If the null hypothesis "it makes no difference which one is right" holds, splitting into 45 and 87 is the same as tossing a coin 132 times.

• exact value from enumerating the binomial (McNemar's test): $p = 3.232\times10^{-4}$
• a permutation test flipping the signs at random, 200,000 times: the observed $|b-c|=42$ was reached or exceeded 68 times, $p = 0.000340$

The two agree. The permutation test uses no normal distribution and no $\sigma$: it just builds "what it would look like if direction meant nothing" 200,000 times and counts. You can write that even when you cannot recall the formula.

How many items does it take to see a difference of 0.01? With disagreement rate $r$ and target difference $d$, $z = d\sqrt{N}/\sqrt{r}$, so

For $r=0.11$ and $d=0.01$ that is 4,226 items. But that is the count at which the expected $z$ is exactly 1.96 — the count that reaches significance half the time. To see it 80% of the time, replace $1.96$ with $1.96+0.84$: 8,634 items. Measured on artificial data:

48.7% at 4,226 and 80.2% at 8,626, as the formula says. Without pairing it takes 16,135 (45.3% at the 16,000 in the table). When you see a comparison table claiming a one-point difference, look at the item count first.

4. By the time you picked it, the score is already up

Next, the drift that comes from looking at the same test set over and over.

Take $k$ candidates with exactly the same ability (true accuracy 0.70 for all of them). Measure each on its own 500 items and keep the best. Repeated 20,000 times:

Columns: the candidate count $k$; the mean score of the one that was picked; how much that is above the true 0.70; the same winner re-measured on a fresh 500 items; and how often the winner was called significantly better than 0.70.