Paper Deep Dive — Attention Is All You Need: What Dropping Recurrence Actually Proved

A close reading of the Transformer paper grounded strictly in its own text: the scaled dot-product equation, why the square root of d_k is there, what the ablations exposed, and the limits the authors themselves flagged.

Attention Is All You Need

From a game of telephone to a round table

"Attention Is All You Need" (Vaswani et al., eight authors from Google Brain, Google Research and elsewhere), posted to arXiv in June 2017, made a blunt proposal: strip recurrence and convolution out of sequence transduction entirely.

At the time the field ran on RNN and CNN encoder-decoders, with the best models connecting encoder and decoder through an attention mechanism (Abstract). An RNN factors computation along symbol positions, producing hidden state $h_t$ at position $t$ from the previous state $h_{t-1}$ and the input at $t$. The paper's objection is precise: this is "inherently sequential" and therefore precludes parallelization within training examples, and since memory constraints limit batching across examples, the problem bites harder as sequences get longer (§1).

Picture an RNN as a game of telephone played in a single line. To move information from one end to the other, it has to pass through everyone in between. The Transformer plays it as a round table instead: everyone sees everyone else's contribution at once and weighs who matters to them.

With that round-table mechanism alone the paper reports 28.4 BLEU on WMT 2014 English-to-German — more than 2 BLEU above the previous best results including ensembles — and 41.8 BLEU on English-to-French, trained for 3.5 days on eight GPUs (Abstract, Table 2). The base model takes twelve hours on eight P100s (§1, §5.2).

The intuition: make the distance between any two positions constant

Why can recurrence go? The background section (§2) puts a finger on it. In ByteNet and ConvS2S, which also set out to reduce sequential computation, the number of operations needed to relate signals from two arbitrary positions grows with the distance between them — linearly for ConvS2S, logarithmically for ByteNet. Distant dependencies are correspondingly hard to learn. In the Transformer that number drops to a constant.

The paper is honest about the price in the same paragraph: averaging attention-weighted positions gives you reduced effective resolution, and Multi-Head Attention is introduced as the counteracting measure (§2). This is not "attention is universally better" — it is an explicit trade: you buy a constant path length, you lose resolution, and you buy the resolution back with multiple heads. That framing gets lost in most retellings.

Section 4 lays out three criteria for preferring self-attention: total computation per layer, the amount of computation that can be parallelized (measured as the minimum number of sequential operations), and the maximum path length between long-range dependencies.

What Table 1 actually shows is that self-attention does not win on complexity unconditionally. As the paper states, self-attention layers are faster than recurrent layers when the sequence length $n$ is smaller than the representation dimensionality $d$, which it argues "is most often the case" for the word-piece and byte-pair sentence representations used by contemporary translation systems (§4). The victory comes with a precondition attached.

The mechanism: scaled dot-product attention

The paper's definition of attention is almost austere: a function mapping a query and a set of key-value pairs to an output, where the output is a weighted sum of the values and each weight comes from a compatibility function of the query with the corresponding key (§3.2).

Then Equation (1) (§3.2.1):

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{QK^{T}}{\sqrt{d_k}}\right)V$$

with $Q$, $K$, $V$ the queries, keys and values packed as matrices and $d_k$ the key dimension.

Read in words: score every query against every key, shrink the scores by $\sqrt{d_k}$, squash them into weights that add up to one, and use those weights to blend the values. What is striking is what the formula does not contain — position. Shuffle the input tokens and the output rows simply shuffle with them, which is why word order has to be injected separately (§3.5, below). If you want the shapes and the projections drawn out step by step, Attention from Scratch covers exactly this equation.

Why divide by $\sqrt{d_k}$? The paper compares against additive attention (which computes compatibility with a one-hidden-layer feed-forward network) and notes that although the two have similar theoretical complexity, dot-product attention "is much faster and more space-efficient in practice, since it can be implemented using highly optimized matrix multiplication code." But for large $d_k$, unscaled dot-product attention loses to additive attention. The paper's explanation lives in footnote 1: if the components of $q$ and $k$ are independent random variables with mean 0 and variance 1, then $q\cdot k=\sum_{i=1}^{d_k}q_i k_i$ has mean 0 and variance $d_k$. Bigger dimension, bigger dot products, and the softmax gets pushed into regions with extremely small gradients. Dividing by the standard deviation $\sqrt{d_k}$ undoes that (§3.2.1). Note the paper's own wording — "We suspect that …" This is offered as a hypothesis, not a proof, and it is worth quoting it that way.