Paper Deep Dive — Attention Is All You Need: What Dropping Recurrence Actually Proved

A close reading of the Transformer paper grounded strictly in its own text: the scaled dot-product equation, why the square root of d_k is there, what the ablations exposed, and the limits the authors themselves flagged.

Attention Is All You Need

From a game of telephone to a round table

"Attention Is All You Need" (Vaswani et al., eight authors from Google Brain, Google Research and elsewhere), posted to arXiv in June 2017, made a blunt proposal: strip recurrence and convolution out of sequence transduction entirely.

At the time the field ran on RNN and CNN encoder-decoders, with the best models connecting encoder and decoder through an attention mechanism (Abstract). An RNN factors computation along symbol positions, producing hidden state $h_t$ at position $t$ from the previous state $h_{t-1}$ and the input at $t$. The paper's objection is precise: this is "inherently sequential" and therefore precludes parallelization within training examples, and since memory constraints limit batching across examples, the problem bites harder as sequences get longer (§1).

Picture an RNN as a game of telephone played in a single line. To move information from one end to the other, it has to pass through everyone in between. The Transformer plays it as a round table instead: everyone sees everyone else's contribution at once and weighs who matters to them.

With that round-table mechanism alone the paper reports 28.4 BLEU on WMT 2014 English-to-German — more than 2 BLEU above the previous best results including ensembles — and 41.8 BLEU on English-to-French, trained for 3.5 days on eight GPUs (Abstract, Table 2). The base model takes twelve hours on eight P100s (§1, §5.2).

The intuition: make the distance between any two positions constant

Why can recurrence go? The background section (§2) puts a finger on it. In ByteNet and ConvS2S, which also set out to reduce sequential computation, the number of operations needed to relate signals from two arbitrary positions grows with the distance between them — linearly for ConvS2S, logarithmically for ByteNet. Distant dependencies are correspondingly hard to learn. In the Transformer that number drops to a constant.

The paper is honest about the price in the same paragraph: averaging attention-weighted positions gives you reduced effective resolution, and Multi-Head Attention is introduced as the counteracting measure (§2). This is not "attention is universally better" — it is an explicit trade: you buy a constant path length, you lose resolution, and you buy the resolution back with multiple heads. That framing gets lost in most retellings.

Section 4 lays out three criteria for preferring self-attention: total computation per layer, the amount of computation that can be parallelized (measured as the minimum number of sequential operations), and the maximum path length between long-range dependencies.

What Table 1 actually shows is that self-attention does not win on complexity unconditionally. As the paper states, self-attention layers are faster than recurrent layers when the sequence length $n$ is smaller than the representation dimensionality $d$, which it argues "is most often the case" for the word-piece and byte-pair sentence representations used by contemporary translation systems (§4). The victory comes with a precondition attached.

The mechanism: scaled dot-product attention

The paper's definition of attention is almost austere: a function mapping a query and a set of key-value pairs to an output, where the output is a weighted sum of the values and each weight comes from a compatibility function of the query with the corresponding key (§3.2).

Then Equation (1) (§3.2.1):

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{QK^{T}}{\sqrt{d_k}}\right)V$$

with $Q$, $K$, $V$ the queries, keys and values packed as matrices and $d_k$ the key dimension.

Read in words: score every query against every key, shrink the scores by $\sqrt{d_k}$, squash them into weights that add up to one, and use those weights to blend the values. What is striking is what the formula does not contain — position. Shuffle the input tokens and the output rows simply shuffle with them, which is why word order has to be injected separately (§3.5, below). If you want the shapes and the projections drawn out step by step, Attention from Scratch covers exactly this equation.

Why divide by $\sqrt{d_k}$? The paper compares against additive attention (which computes compatibility with a one-hidden-layer feed-forward network) and notes that although the two have similar theoretical complexity, dot-product attention "is much faster and more space-efficient in practice, since it can be implemented using highly optimized matrix multiplication code." But for large $d_k$, unscaled dot-product attention loses to additive attention. The paper's explanation lives in footnote 1: if the components of $q$ and $k$ are independent random variables with mean 0 and variance 1, then $q\cdot k=\sum_{i=1}^{d_k}q_i k_i$ has mean 0 and variance $d_k$. Bigger dimension, bigger dot products, and the softmax gets pushed into regions with extremely small gradients. Dividing by the standard deviation $\sqrt{d_k}$ undoes that (§3.2.1). Note the paper's own wording — "We suspect that …" This is offered as a hypothesis, not a proof, and it is worth quoting it that way.

Multi-head attention (§3.2.2) replaces one attention computation over $d_{\text{model}}$-dimensional inputs with $h$ learned linear projections, attention computed in parallel on each, then concatenation and one more projection:

$$\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}(\mathrm{head}_1,...,\mathrm{head}_h)W^{O}$$

The first line is a recipe, which says: run the whole attention computation $h$ separate times, lay the $h$ outputs side by side, and multiply the concatenation by one more learned matrix $W^{O}$ to blend them back into a single output.

$$\mathrm{head}_i=\mathrm{Attention}(QW^{Q}_i,KW^{K}_i,VW^{V}_i)$$

The second line says what one of those runs is: the same Equation (1) as before, except that $Q$, $K$ and $V$ are each squeezed through that head's own projection first. Every head therefore looks at the same tokens through a different, narrower window.

The stated rationale: it "allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this." The paper uses $h=8$ and $d_k=d_v=d_{\text{model}}/h=64$, and points out that because each head's dimension is divided down, the total computational cost is similar to single-head attention at full dimensionality (§3.2.2). Adding heads is not the same as making the model heavier.

The rest of the stack is equally plain. The encoder is $N=6$ identical layers, each with two sub-layers — multi-head self-attention and a position-wise fully connected feed-forward network — each wrapped in a residual connection and layer normalization, $\mathrm{LayerNorm}(x+\mathrm{Sublayer}(x))$, with $d_{\text{model}}=512$ (§3.1). The decoder adds a third sub-layer attending over the encoder output, and masks illegal connections in its self-attention by setting them to $-\infty$ in the softmax input (§3.1, §3.2.3).

The feed-forward network has inner dimension $d_{ff}=2048$ and is described as "two convolutions with kernel size 1" (§3.3, Equation (2)):

$$\mathrm{FFN}(x)=\max(0,xW_1+b_1)W_2+b_2$$

Again in words: take each position's vector on its own, expand it to 2048 dimensions, clamp everything negative to zero, and project it back down to 512. The same two matrices are reused at every position, and no position ever consults another one here — mixing across positions is the attention sub-layer's job, and this layer only reshapes each position independently.

Position information arrives as sinusoidal positional encodings added to the embeddings, with wavelengths forming a geometric progression from $2\pi$ to $10000\cdot 2\pi$. The reason for that choice is again framed as a hypothesis: for any fixed offset $k$, $PE_{pos+k}$ can be expressed as a linear function of $PE_{pos}$, so the model should find it easy to learn to attend by relative position (§3.5). The route from this absolute, added-in encoding to the relative schemes and RoPE that later displaced it is traced in Positional Encoding from Scratch.

What the experiments showed — and what they didn't

Training used roughly 4.5M sentence pairs for English-German (BPE, shared source-target vocabulary of about 37000 tokens) and 36M sentences for English-French (32000 word-piece), batched to roughly 25000 source and 25000 target tokens (§5.1). Optimization was Adam with $\beta_1=0.9$, $\beta_2=0.98$, $\epsilon=10^{-9}$, learning rate ramped linearly for 4000 warmup steps then decayed by the inverse square root of the step number (§5.3, Equation (3)). Regularization: residual dropout $P_{drop}=0.1$ and label smoothing $\epsilon_{ls}=0.1$ (§5.4).

Table 2 gives base 27.3 (EN-DE) / 38.1 (EN-FR) and big 28.4 / 41.8, at estimated training costs of $3.3\cdot 10^{18}$ and $2.3\cdot 10^{19}$ FLOPs versus, for example, GNMT+RL at $2.3\cdot 10^{19}$ / $1.4\cdot 10^{20}$. Footnote 2 is explicit that these FLOPs are an estimate, obtained by multiplying training time, GPU count, and an assumed sustained single-precision throughput per GPU. One inconsistency worth flagging: the body of §6.1 reports the big EN-FR model at "41.0 BLEU," which disagrees with the 41.8 in both the abstract and Table 2, and the discrepancy survives in v7.

The ablations (§6.2, Table 3) are where it gets interesting.

• Head count: a single head is 0.9 BLEU worse than the best setting — but quality also drops with too many heads ($h=32$ gives BLEU 25.4 against the base model's 25.8).
• Shrinking the key dimension $d_k$ hurts quality, and the paper draws a pointed conclusion about its own design: "determining compatibility is not easy and … a more sophisticated compatibility function than dot product may be beneficial."
• Swapping sinusoids for learned positional embeddings changes almost nothing (PPL 4.92 / BLEU 25.7 versus the base 4.92 / 25.8, row (E)). Sinusoids were kept because they may allow extrapolation to sequences longer than those seen in training — an expectation the paper does not test (§3.5).

As a generalization check, a 4-layer model with $d_{\text{model}}=1024$ was trained on English constituency parsing: F1 91.3 using only the ~40K WSJ sentences, and 92.7 in a semi-supervised setting with roughly 17M sentences (§6.3, Table 4). With almost no task-specific tuning it beat every previously reported result except the Recurrent Neural Network Grammar (93.3).

Limitations and open questions

Acknowledged by the paper itself:

• Quadratic in sequence length. At $O(n^2\cdot d)$, self-attention beats recurrence only when $n<d$ (§4, Table 1). Restricting attention to a neighborhood of size $r$ is floated for very long sequences, but explicitly as future work — it is not evaluated here (§4).
• Reduced effective resolution from averaging attention-weighted positions (§2). Multi-head attention mitigates it; it does not remove it.
• The dot product may be too weak a compatibility function, as ablation row (B) suggests (§6.2).
• Narrow evaluation: two translation tasks and one English parsing task. Non-text modalities (images, audio, video), local/restricted attention, and making generation less sequential are all listed in the conclusion as things the authors plan to do (§7). The image half of that list is the one that got answered most directly; Paper Deep Dive — ViT covers how.
• The interpretability claim is soft. The appendix visualizations show heads in layer 5 of 6 that appear to track the long-distance dependency in "making … more difficult" and that seem involved in resolving "its" (Figures 3–5). The paper's own hedging language is "appear to" and "seems related." There is no quantitative evaluation.

Note as of this writing in 2026 (this is the author's annotation, not a claim from the paper): the $O(n^2)$ cost and the $n<d$ precondition turned out to be the paper's most binding constraints once long contexts became the norm. Long-context handling, positional representation design, and encoder-decoder versus other configurations have all moved on, and today's implementations are not this architecture verbatim. Every number in this article is from this paper; none come from later work.

Practical takeaways

The following are the author's operational readings of what the paper reports.

• If you write attention by hand, don't drop the $1/\sqrt{d_k}$. The variance of the dot product scales with $d_k$, and the softmax saturates (§3.2.1, footnote 1). It is the first thing to check when a hand-rolled attention layer refuses to train.
• More heads is not simply better. Table 3(A) shows one head is clearly worse, and 32 heads is worse than the base configuration too.
• Cutting $d_k$ is a tempting saving that costs quality (Table 3(B)). Look elsewhere for cheap wins.
• Part of the headline score is post-processing. The paper averages the last 5 checkpoints for base models and the last 20 for big ones, and decodes with beam size 4 and length penalty $\alpha=0.6$ (§6.1). Comparisons against other methods shift if this isn't matched.
• Don't judge on validation perplexity alone. Label smoothing at 0.1 is stated to hurt perplexity while improving accuracy and BLEU (§5.4).
• Don't agonize over positional representation — at least not at this scale of translation task. Learned embeddings and sinusoids performed essentially the same (Table 3(E)).

How this shows up at work: the paper's hyperparameters in practice

You rarely re-derive this paper. You meet it as a config file. Anyone porting a research implementation, sizing a training run, or debugging an attention layer that will not converge is reading §3 whether they know it or not, because the names in a modern model config map one-to-one onto it: hidden_size is $d_{\text{model}}$, num_attention_heads is $h$, intermediate_size is $d_{ff}$, and the ratio between the first two is $d_k$.

Where that mapping bites:

• The head-division constraint is a runtime error, not a design note. Because $d_k = d_{\text{model}}/h$ (§3.2.2), torch.nn.MultiheadAttention refuses an embed_dim that is not divisible by num_heads. Widening a model from 512 to 768 while leaving heads at 8 is fine; going to 500 is not.
• The scale factor is already there — don't apply it twice. torch.nn.functional.scaled_dot_product_attention defaults its scale argument to $1/\sqrt{d_k}$. Hand-rolled code that divides the scores and then calls a fused kernel that divides again produces a distribution far too flat to learn from, and it fails silently: no error, just a loss curve that flattens out early.
• A forgotten causal mask is the classic silent failure. §3.2.3 masks illegal connections to $-\infty$ before the softmax; is_causal or an explicit attn_mask is where that lives in practice. Omit it and the decoder attends to the future — teacher-forced validation loss looks excellent and generation is nonsense, because the leak exists only at training time.
• Reproducing a BLEU number means reproducing the decode, not just the model. Checkpoint averaging (5 for base, 20 for big), beam size 4, length penalty $\alpha=0.6$ (§6.1) and tokenization all move the score. A gap against a published number is more often a decoding-config gap than a modelling one, so check num_beams and length_penalty before rewriting the architecture.
• The $n<d$ precondition expires quietly. Table 1's advantage over recurrence holds when sequence length is below the representation dimension (§4). Nothing warns you when a context-length increase crosses that line; the $O(n^2 \cdot d)$ term simply starts dominating the step time and the memory profile.

The interview version of this paper is almost always the same question — why the $\sqrt{d_k}$? The answer to give is the chain, not the fact: with unit-variance components the dot product has variance $d_k$, so larger $d_k$ means larger logits, which saturates the softmax into a region of vanishing gradients; dividing by the standard deviation undoes it. Adding that the paper itself only suspects this (footnote 1) is the part that shows you read the source rather than a summary of it.

Summary

• The claim is not "attention alone is sufficient" so much as a trade: you gain a distance-independent constant path length, you lose effective resolution, and multi-head attention buys it back (§2).
• There are really only two equations, (1) and (2). The subtlety is concentrated in why $\sqrt{d_k}$ appears (§3.2.1, footnote 1) and why splitting into heads doesn't increase cost (§3.2.2).
• The paper documents its own weak points: $O(n^2)$, effective resolution, the limits of a dot-product compatibility function, and a narrow evaluation set (§4, §6.2, §7).

Original paper: Vaswani et al., "Attention Is All You Need" (arXiv:1706.03762, 2017)