Attention from Scratch — The Heart of the Transformer, Explained Visually

Self-attention, the core mechanism behind ChatGPT, explained from zero: analogy, intuition, matrix mechanics, and runnable numpy code. No math background required.

Why do we need "attention" at all?

Consider the word bank. In "I opened a bank account" it means a financial institution; in "I sat on the river bank" it means the edge of a river. The word itself doesn't carry its meaning — the surrounding context does.

Older neural networks (RNNs) read a sentence left to right, one word at a time, squeezing everything they had seen into a single memory vector. The longer the sentence, the more the early words faded — essentially a game of telephone.

That design has two separate weaknesses. The first is distance: to connect two far-apart words, the signal has to survive one processing step for every word in between, so the link gets weaker the further apart they are. The second is speed: word two cannot be processed until word one is finished, so the words of a sentence cannot be handled at the same time. A GPU is at its best when it can run one big identical calculation across everything at once, and this architecture gave it no way to do that.

Attention changed this at the root. Every word looks at every other word in the sentence simultaneously and assigns a weight to how relevant each one is for determining its own meaning. The 2017 paper "Attention Is All You Need" showed that this mechanism alone is enough to build a language model, and named the architecture the Transformer. The "T" in ChatGPT comes from here.

An analogy: gathering information in a meeting

Picture self-attention as a meeting. You (one word) are taking notes for a summary.

1. You hold a Query — "here's the kind of information I'm looking for"
2. Every participant wears a Key — a name tag saying "here's the kind of information I have"
3. You compare your query against each name tag, listen closely to relevant people, and tune out the rest
4. Your final note is a blend of everyone's Value — what they actually said — mixed in proportion to how much you listened

Those are the three roles: Query, Key, Value. Self-attention is every word performing this "ask, match, blend" routine at the same time.

The part worth holding on to is that all three are different views derived from the same word. The same token says "here is what I'm looking for" as a query, "here is what I hold" as a key, and "here is what I hand over" as a value. Because the roles are separated, "how much A cares about B" and "how much B cares about A" come out as two different numbers — much as in a real meeting, where what you want to hear and what you have to offer rarely line up.

Intuition: it's just a weighted average

Before the math scares you off, here's the punchline: the output of self-attention is a weighted average of the Value vectors.

• The weights come from "how similar is my Query to each Key?"
• Similarity is measured with the dot product (multiply element-wise, sum up)
• The weights are normalized with softmax so they add up to 1

If you remember one thing, make it this: similar vectors have a large dot product. That's the whole trick.

A large dot product means the two vectors point in roughly the same direction. How words get turned into directional lists of numbers in the first place is the subject of Embeddings from Scratch; the only thing you need to carry into this article is one line: words with related meanings are trained to point in similar directions. Attention is the machinery that converts that closeness of direction into a share of "how much I listen to you".

The mechanics: one big matrix multiplication

Each word is first converted into an embedding vector (say, 512 numbers). A 6-word sentence becomes a 6×512 matrix X.

Self-attention uses three learnable weight matrices W_q, W_k, W_v to derive three matrices from X:

Q = X @ W_q   # (6, 64)  each row is a word's "question"
K = X @ W_k   # (6, 64)  each row is a word's "name tag"
V = X @ W_v   # (6, 64)  each row is a word's "content"

Then all query–key dot products are computed in a single matrix multiply:

scores = Q @ K.T   # (6, 6)  scores[i][j] = how much word i cares about word j

This 6×6 table is the blueprint of attention. Read row i and you can see exactly which words word i is looking at.

Reading it takes a small knack. Rows are the lookers, columns are the looked-at. A large value at row 3, column 2 means "the third word is strongly referring to the second". And because the normalization runs along each row, every row sums to exactly 1 while the columns do not. Some words get referenced by many others; some get looked at by nobody. The table is square, but it is not symmetric.

Putting the whole thing into one line gives the Scaled Dot-Product Attention from the paper:

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$

It looks intimidating, but it is exactly the three steps above: $QK^\top$ is the query-vs-key score table, dividing by $\sqrt{d_k}$ (where $d_k$ is the key dimension, 64 in our example) keeps the dot products from saturating the softmax as dimensions grow, and multiplying by $V$ takes the weighted average of what each word has to say.

Strip the notation and you are left with a single sentence, which says: compare everyone's question against everyone's name tag to build a table of scores, calm those scores down with a division, turn them into proportions, then blend what everyone said in exactly those proportions. Four symbols in a row, and the work being done is note-taking in a meeting.

Self-attention in code (numpy pseudo-code)

The whole thing fits in a few lines:

import numpy as np

def self_attention(X, W_q, W_k, W_v):
    Q = X @ W_q
    K = X @ W_k
    V = X @ W_v

    d_k = K.shape[-1]                      # key dimension (e.g. 64)
    scores = Q @ K.T / np.sqrt(d_k)        # scaled dot product

    weights = softmax(scores, axis=-1)     # each row sums to 1
    return weights @ V                     # weighted average of Values

def softmax(x, axis=-1):
    e = np.exp(x - x.max(axis=axis, keepdims=True))
    return e / e.sum(axis=axis, keepdims=True)

Dividing by np.sqrt(d_k) matters. With high-dimensional vectors, dot products grow large, pushing softmax into an extreme distribution that attends to essentially one word — which makes training hard. The scaling prevents that, and the paper calls the full recipe Scaled Dot-Product Attention.

There is also one step in the code that appears in no equation: subtracting x.max(...) inside softmax. The exponential blows up fast, so feeding raw scores straight into np.exp overflows into inf and then nan. Subtracting the row maximum first multiplies numerator and denominator by the same constant, so the result is unchanged and the numbers stay in range. Present in every implementation, absent from every formula — this kind of numerical guard rail shows up repeatedly around Transformers.

Multi-head: several perspectives at once

A single QKV set can only capture one kind of relationship. Real Transformers run several smaller QKV sets in parallel (e.g. 8 heads) and concatenate the results:

heads = [self_attention(X, Wq[i], Wk[i], Wv[i]) for i in range(8)]
out = np.concatenate(heads, axis=-1) @ W_o   # concatenate, then mix

In practice, heads naturally specialize — one may track grammatical dependencies while another resolves what a pronoun refers to.

The dimension bookkeeping is worth getting straight. With a 512-dimensional embedding and 8 heads, each head works in 64 dimensions. Concatenating the 8 outputs of 64 dimensions each gets you back to 512, and W_o mixes them before the result moves on. Input and output have the same shape, which is precisely what lets identical blocks stack as deep as you like. And adding heads barely changes the total compute, because each one gets a narrower slice; what grows is the number of distinct relationships the layer can tell apart at once.

Three things that matter in practice

1. Causal masking Generative models like GPT must not "cheat" by peeking at future words. The upper-right triangle of the score matrix is filled with -inf before softmax, so each word can only attend to words before it.

It has to be -inf rather than 0, because the mask is applied before softmax. Softmax exponentiates and then normalizes, so $e^{-\infty}=0$ drives the weight to exactly zero. Fill those cells with 0 instead and they become "candidates that scored zero" — after normalization a small weight survives, and the model gets a faint look at the future. Training loss still falls neatly; the damage only surfaces when you actually generate. It is a genuinely easy bug to miss.

2. Cost grows with the square of sequence length The score matrix is (length × length), so doubling the context quadruples compute and memory. This is exactly why long-context models are expensive, and why optimizations like FlashAttention target this step.

Numbers make it concrete: a 1,000-token sequence gives a table of a million cells, and 10,000 tokens gives a hundred million. Ten times the length, a hundred times the table. What makes it worse is that the table exists in memory partway through the computation — the intermediate result ends up far larger than the output it produces. Methods like FlashAttention attack precisely that, working in small blocks so the full table is never materialized.

3. The KV cache During generation, the K and V of past tokens never change, so they can be reused. That's the KV cache — the classic LLM serving trade-off of faster inference in exchange for more memory.

The detail that pays off later: only K and V go in the cache — Q does not. The query is only ever needed for the single token being generated right now, whereas the keys and values of every past token must be kept around. So memory grows with context length and with the number of requests served concurrently, and that is what puts a ceiling on how many sequences fit on a GPU. The KV Cache from Scratch works through how to estimate that number.

Takeaways

• Self-attention is "a weighted average of Values, with weights from Query–Key dot products"
• Every word sees every word at once, so long-range dependencies are reached in a single step
• Scaling, masking, and multi-head are practical reinforcements on top of that simple core

For how this mechanism is actually presented in the original paper — and which ablations (removing one component at a time to measure its effect) were used to justify the design — see Paper Deep Dive — Attention Is All You Need, which reads it closely against its own text.

Next up: stacking attention into a full Transformer block, including residual connections and LayerNorm.