The Transformer, End to End — One Token's Journey from Embedding to Output

Between typing a prompt and getting a word back, a single token gets handed a vector, rewritten in room after room, and finally turned back into language. Tokenizing, embedding, position, attention, feed-forward, residuals, and the output head — walked as one continuous trip rather than a pile of parts.

Attention Is All You Need

How to read this: follow one token

Almost every Transformer explainer is organized by part. An article on attention. An article on positional encoding. An article on normalization. Each one makes sense on its own, and yet the question "so where exactly does this piece bolt on?" never quite gets answered. It's like being handed the spec sheet for every screw and no assembly diagram.

This article does the opposite. We pick a single token and walk it from the entrance to the exit.

Think of the token as a traveler carrying one notebook. The notebook holds a few hundred numbers. The traveler passes through a series of rooms, and each room edits the notebook a little. Crucially, the notebook is exactly the same size leaving a room as it was entering — which is why you can chain identical rooms indefinitely. And by the time the traveler leaves the last room, that notebook has become a pointer to whatever word should come next.

That's the whole architecture. Everything else is about what happens inside each room.

Stop 0: chopping the string up

Models don't read characters. They read token IDs — a sequence of integers.

The string "the weather is nice" gets matched against a vocabulary and turned into something like [1284, 402, 9931]. The tokenizer decides where the cuts fall, and the pieces are usually subwords: not whole words, not single characters, something in between.

There's one thing to hold onto here. At this point a token is nothing but a jersey number, carrying no meaning at all. Nothing guarantees that number 1284 and number 1285 mean anything alike. The logic behind how the cuts get chosen is covered in Tokenizers from Scratch.

Stop 1: turning a number into a coordinate

The first room that gives those numbers meaning is the embedding layer.

All it does is pull one row out of a very large table. The table is "vocabulary size × model dimension." With a 50,000-word vocabulary and a model dimension of 768, the table is 50,000 rows by 768 columns, and token 1284 walks away with the 768 numbers on row 1284.

Those 768 numbers are the notebook from the opening analogy. From here to the exit, the traveler carries this one notebook and nothing else, and every room rewrites its contents. The vector being passed along across layers like this is often called the residual stream.

The important part is that this table isn't a fixed dictionary — it's learned. As training proceeds, words that appear in similar contexts drift toward similar directions. Nothing has looked at context yet, though, so bank starts life as a single vector with "financial institution" and "river edge" still blended together. Untangling that is the job of everything downstream.

Stop 2: writing in the order

A bare Transformer has a property that looks fatal at first glance: it has no idea what order the input came in.

Attention lets every token see every token, so shuffling the input just permutes the correspondences — nowhere is there a record of which token came first. A model that can't tell "dog bites man" from "man bites dog" is not a language model.

So position gets injected explicitly. The original paper built position vectors out of sines and cosines and added them to the embeddings. Most modern LLMs don't add anything; they rotate the Query and Key vectors just before the attention computation, a scheme called RoPE. Both aim at the same thing — stamping "which slot am I in," or "how far apart are we," into the vectors. Positional Encoding from Scratch works through the details.

The traveler is now packed: the notebook holds an initial guess at meaning, plus a record of where it sits in the sequence.

Stop 3: inside a block — two rooms make a set

A Transformer block contains two sub-rooms: one where the token looks around, and one where it thinks on its own. That pairing is the heart of the design.

First half: self-attention, looking around

Self-attention lets each token survey every token in the sequence, decide how much each one matters for pinning down its own meaning, and blend in their information in those proportions. bank collapses into "financial institution" precisely here — in the room where it picks up a strong signal from account.

Each token derives three views from its own notebook: a Query for what it's looking for, a Key for what it holds, and a Value for what it actually hands over. Pairs whose Query and Key have a large dot product get linked strongly, and those strengths become the mixing proportions for everyone's Values.

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$

In words: build a score table by checking everyone's question against everyone's name tag, convert the scores into proportions, and mix everyone's contents in those proportions. ($d_k$ is the Key dimension; dividing by $\sqrt{d_k}$ keeps the scores from blowing up.) The derivation, plus what multi-head and causal masking actually buy you, is in Attention from Scratch.

One line summarizes this room: it is the only place where information moves between tokens. Every other room in the building processes tokens independently.

Second half: feed-forward, thinking alone

The notebook leaving attention goes into the feed-forward network (FFN). This room never looks at another token. One token's notebook goes in, one token's notebook comes out.

Which says: stretch the vector out into a wider space, bend it with a nonlinearity $\sigma$, then fold it back down to the original width. ($W_1, W_2$ are learned matrices, $b_1, b_2$ are biases, and $\sigma$ is an activation such as ReLU.) Four times the model dimension is the conventional stretch — the original paper widened 512 dimensions to 2048 before folding back.

It looks like the boring room, but it holds most of the block's parameters. With model dimension $d$, the attention side carries four weight matrices (Q, K, V, and the output projection) for $4d^2$ parameters total, while the FFN carries two $d \times 4d$ matrices for $8d^2$. Roughly two-thirds of a block's weights live in the FFN. When people say knowledge is stored in the FFN, part of what they're pointing at is sheer volume.

House rules on the way out: residuals and normalization

A sub-room's output doesn't simply get passed along. It is added back onto the notebook that walked in, with a normalization step in the loop.

In words, equation (2) says a room never replaces the notebook — it only writes a correction into it. That addition is the residual connection, and it's what lets gradients reach the bottom of a deep stack without the training falling apart. $\mathrm{Norm}$ rescales the numbers to a consistent magnitude; the original paper placed it after each room, but putting it before (Pre-LN) trains more stably in deep models and has become the default.

The same room, dozens of times

This is where the architecture is most often misread. There is only one kind of Transformer block. The model just stacks it dozens of times.

Because the notebook is the same size going in and coming out, blocks chain freely. But the weights are not shared: layer 1's FFN and layer 2's FFN are entirely different matrices. Same floor plan, completely different furniture.

As the stack progresses, the notebook shifts in character — from "what this word means" toward "what role this position plays in this particular sentence."

The exit: turning a vector back into words

Leaving the final block, the notebook is still just a few hundred numbers. The output head converts it back into language.

It's the mirror image of the embedding step: multiply by a "model dimension × vocabulary size" matrix to produce one score (a logit) for every word in the vocabulary. Some implementations reuse the transposed embedding table for this matrix — weight tying. Then softmax turns those scores into probabilities.

In words: take the final-layer notebook $h_L$, turn it into vocabulary scores $z$, divide by a temperature $T$, and normalize into proportions. Shrink $T$ and the top-scoring word wins almost every time; raise it and the distribution flattens until rare words start appearing. The temperature parameter in every inference API is exactly this $T$.

A generative model then samples one token from that distribution, appends it to the input, and sends a new traveler out from Stop 0. One full trip produces one token of output.

The whole thing as code

Written as pseudocode, all of the above is startlingly short.

def transformer(ids):
    h = embed[ids] + pos(len(ids))         # stops 1 and 2
    for blk in blocks:                     # the same room, N times
        h = h + attn(norm1(h), blk)        # first half: look around
        h = h + ffn(norm2(h), blk)         # second half: think alone
    return norm_f(h) @ W_u.T               # exit: vocabulary logits

Five lines — and a whole shelf of articles written about the parts inside them. Which is the point: the difficulty of the Transformer isn't in its structure, it's in what happens inside each room.

A shape cheat sheet

What actually helps when debugging is knowing what dimension you're holding at each step. With sequence length $n$, model dimension $d$, and vocabulary size $V$:

Two rows are fat. The score table grows with the square of sequence length, and the logits grow with vocabulary size. Why long contexts are expensive, and why a bigger vocabulary makes the output layer heavier, both fall straight out of this table.

Compute cost follows from the same shapes. Attention scales as $n^2 d$; the FFN scales as $n d^2$. For short sequences the FFN dominates, and once $n$ grows well past $d$, attention takes over. That handoff is what's really happening when a model "suddenly gets slow" on long contexts.

How this shows up on the job

Reading a config file. Open any Hugging Face model and you'll end up in config.json, where the fields map one-to-one onto the stops above. vocab_size is the vocabulary from Stop 0, hidden_size is the thickness $d$ of the notebook, num_hidden_layers is how many times you go through the room, num_attention_heads is how many viewpoints attention gets, intermediate_size is how wide the FFN stretches (usually around 4× hidden_size), and max_position_embeddings is the longest sequence Stop 2 was built for. Read those six and you can ballpark the model's size and memory footprint.

Choosing what to fine-tune. When you set target_modules for LoRA, this map is your menu. q_proj, k_proj, v_proj, and o_proj are the four attention matrices; gate_proj, up_proj, and down_proj are the FFN. Adapting attention only is the lightweight default, but as noted, the parameter volume sits in the FFN — so if you're trying to move knowledge rather than routing, including the FFN is a defensible call. Which one wins is task-dependent, and the honest answer in practice is to run both and compare.

Tuning inference. temperature is the $T$ at the exit; top_p and top_k trim the candidate set after the softmax. A recurring support ticket here is "I set temperature=0 and the output still varies." Most implementations replace temperature 0 with straight argmax, but near-ties can still flip when floating-point reduction order changes across parallel hardware, so bit-exact reproducibility is not guaranteed. If reproducibility is a requirement, pin the seed and the implementation, then verify it.

Three traps worth knowing about. First, input longer than max_position_embeddings pushes position handling outside its training range; this often doesn't raise an error, it just quietly degrades quality, so check that number before feeding in long documents. Second, when you extend a vocabulary you have to widen both the embedding table at Stop 1 and the output matrix at the exit — with weight tying one edit covers both, but without it, widening only one side gives you a dimension mismatch. Third, don't reach for the final layer's output when you want a sentence embedding: the last layer is optimized to predict the next token, which is not the same objective as representing the meaning of the whole sequence. Compare it against intermediate layers and other pooling strategies before committing.

How this gets asked in reviews and interviews. "If you double the sequence length, how much more compute?" — the cheat sheet answers it directly. The attention term goes up 4×, the FFN term goes up 2×, and which one dominates depends on the ratio of $n$ to $d$. Get that far and you can also explain, on the same map, exactly what KV caching and FlashAttention are each trying to cut.

Summary

• A token is one notebook: initialized by the embedding table, stamped with position, then edited by a correction in every room it passes through
• A block holds two rooms. Attention mixes across tokens; the FFN transforms within a token. Attention is the only place mixing happens
• Residual connections make each room additive rather than substitutive, which is what makes deep stacks trainable
• At the exit the vector returns to vocabulary-sized scores, and a temperature-scaled softmax turns them into probabilities
• Two places get fat: the attention score table (squared in sequence length) and the output logits (linear in vocabulary size)

Each stop on this map has its own dedicated article. Going back to those part-by-part pieces with the whole route in your head tends to make them read very differently.