Chapter 8 was the training budget, Chapter 9 the stages that fit it to people, Chapter 10 the machinery for pulling things in from outside. All of them are about how it gets built.

This chapter counts what gets paid when it is used. Training happens once; inference happens on every response. Knowing where the price is attached also shows that there are only two ways to make it faster.

1. The price of one token, and when inference overtakes training

When Chapter 8 counted $C \approx 6ND$, the breakdown was $2N$ forward and $4N$ backward. Inference is forward only, so per token

At $N = 7\times10^{9}$ that is $1.4\times10^{10}$ FLOPs per token, and $1.4\times10^{13}$ for a 1,000-token response.

Training is $6ND$, once. So inference catches up with training after, for responses of $L$ tokens,

responses. The interesting part is that $N$ cancels: it does not depend on model size. With $D = 10^{12}$,

2. Keeping it so it is not rebuilt — the KV cache

Autoregression emits one token at a time. Done naively, producing token $t$ means recomputing everything from the start.

The waste is countable. To emit $T$ tokens, running $t$ tokens' worth of forward pass at step $t$ totals $T(T+1)/2$; keeping the past around makes it $T$.

So the $K$ and $V$ of past tokens (Chapter 6) are kept. That is the KV cache, and it is paid for in memory: two tensors, per layer, per token, $d$ wide, $b$ bytes each —

Per token, at 32 layers, $d$=4,096, 2 bytes, that is 0.524 MB. Serving eight people at once at $T$=8,192 needs 34.4 GB for the cache alone.

3. What sets the speed is reading, not computing

Producing one token means reading every weight once. $N$ weights at $b$ bytes each is $Nb$ bytes read for $2N$ FLOPs of work. Divide:

At 16 bits (2 bytes) that is 1 FLOP/byte — one multiply per byte read.

Running $B$ requests together means the weights are read once for all of them:

Which of reading and computing runs out first on real hardware depends on the machine. What this book can give is the ratio itself; which side is the bottleneck on which machine is not measured here.

4. Reading less — quantization

Round the weights to $b$ bits. The plainest way: take the largest magnitude $m$ in some range and split it into $2^{b-1}-1$ even steps.

If the rounding error spreads uniformly over $\pm\text{step}/2$, its variance is $\text{step}^2/12$. Checked on standard normal weights, 4,096 per row, 64 rows.

Eight through three bits agree within 0.5%; only 2 bits breaks away. The reason is measurable too. At 2 bits the grid has three rungs, $\{-m, 0, +m\}$, and 93.9% of the weights round to zero. The assumption that error spreads uniformly is what fails first.

Who shares a scale

The table above took one maximum across all 4,096 values in a row. Taking it again every 128 values:

A steady 1.34x improvement at every bit width. So far this is a modest gain.

When one outlier gets in

Put a single weight of $\pm20$ in each row and it changes character.