Chapter 5 looked at two assumptions. Convolution says "only nearby things relate"; recurrence says "process in order." In both, the structure decides in advance where to look — inside the window, or one step back. Whatever the input, the place being looked at does not change.

Attention lets the input decide. Drop one assumption, and compute "where to look" instead. This chapter measures what that buys and what it costs.

1. Build a table of "where does it look"

For a sequence of length $T$, consider the $T \times T$ table of how much position $i$ attends to position $j$.

• Convolution: that table is a fixed band. Values sit only where $|i-j| \le 1$, identically for every input.
• Attention: that table is computed from the input.

The figure measures how far the table moves when the content changes. The convolution is always 0: it sees only position, so whatever the content becomes, the table is the same. Attention moves.

Note that this is all it shows. In this untrained figure there is no guarantee that matching symbols attract — measured, some draws make them lower. "The table changes with the content" is a property of the structure; "it attends to what matters" is a result of training. Only the former can be demonstrated here, and the latter cannot be read off weights at all — which the final section returns to.

2. The formula — splitting into three roles

All three come from the same input:

• Query $Q$ — "what am I looking for"
• Key $K$ — "what do I show up as"
• Value $V$ — "what do I hand over when found"

$QK^{\top}$ is the inner product of what is sought with what is findable — a similarity (Chapter 0-1, equation (1)). Softmax turns it into weights summing to one (the categorical distribution of Chapter 0-3), and those weights mix $V$.

Building the seeking side and the findable side with separate matrices is one of the points. Use the same vector for both and "most similar to itself" always holds, lighting up only the diagonal of the table.

3. Why divide by $\sqrt{d_k}$ — measured

The denominator in equation (1) can be left out and the code still runs. It runs, and training stalls. The reason is variance.

If every component of $q, k$ has mean 0 and variance 1, the inner product $q \cdot k$ has variance $d_k$ (variances add when independent — Chapter 0-3). Measured:

The spread grows with the dimension. Feed that spread into a softmax and only the largest survives. Measured with 16 keys:

(Uniform would be $1/16 = 0.0625$ and 4.0 bits.)

Undivided at $d_k = 512$, the entropy is 0.0000 bits — one of sixteen, completely. There the softmax gradient is also ~0 (the flat region from Chapter 0-2), so where to look cannot be learned. Dividing by $\sqrt{d_k}$ leaves 3.60 bits, and room to learn.