How Transformers Work
Attention, positional encoding, and the architecture dissected
01
·How Transformers Work·FREE·8 min read
Attention from Scratch — The Heart of the Transformer, Explained Visually
Self-attention, the core mechanism behind ChatGPT, explained from zero: analogy, intuition, matrix mechanics, and runnable numpy code. No math background required.
02
·How Transformers Work·FREE·PAPER·10 min read
Positional Encoding from Scratch — From Absolute Positions to RoPE
A bare Transformer has no idea what word order is. Starting from why position information is needed at all, this article walks through sinusoidal absolute encodings, learned embeddings, and RoPE — the modern LLM standard — showing exactly why rotation encodes relative position.
03
·How Transformers Work·FREE·PAPER·10 min read
Tokenizers from Scratch — The Unit an LLM Cuts the World Into
An LLM reads neither characters nor words. How BPE builds a vocabulary, what SentencePiece actually fixed, why some languages pay more for the same sentence, and what you trade away when you grow the vocabulary — worked by hand and in code, from zero.
04
·How Transformers Work·★ MEMBER·PAPER·9 min read
A Field Guide to Attention Variants — MQA, GQA, Sliding Windows, Linear Attention
MQA, GQA, sliding windows and linear attention are not four unrelated tricks. One multiplication decides how large a KV cache gets, and every variant is a decision about which factor in it to attack. Lined up against that formula, the family tree shows exactly what each one gave up and what it bought.
05
·How Transformers Work·FREE·PAPER·10 min read
The Transformer, End to End — One Token's Journey from Embedding to Output
Between typing a prompt and getting a word back, a single token gets handed a vector, rewritten in room after room, and finally turned back into language. Tokenizing, embedding, position, attention, feed-forward, residuals, and the output head — walked as one continuous trip rather than a pile of parts.
06
·How Transformers Work·★ MEMBER·PAPER·10 min read
FlashAttention from Scratch — The Paradox of Doing More Math to Go Faster
FlashAttention deliberately recomputes the same math in the backward pass. It does strictly more arithmetic and still runs faster — because on a GPU, moving numbers costs more than multiplying them. From counting HBM round trips, through the softmax wall that blocks tiling, to the online-softmax recurrence that breaks it, and the silent backend fallback that bites people in production.
07
·How Transformers Work·★ MEMBER·PAPER·11 min read
How Long-Context LLMs Work — From RoPE Interpolation to Ring Attention
A "128K context window" is two different walls, knocked down by two unrelated families of tricks. This walks through positional interpolation, NTK-aware scaling and YaRN for the position wall; sliding windows and ring attention for the compute wall; and how to read a needle-in-a-haystack chart without being fooled by it.
08
·How Transformers Work·★ MEMBER·PAPER·9 min read
Encoder or Decoder — The Fork in the Road Between BERT and GPT
One masked triangle in the attention table is what separated BERT from GPT. This piece works through bidirectional versus autoregressive with the equations and an interactive figure, then asks why generation won and where encoders are still the first choice.
09
·How Transformers Work·★ MEMBER·PAPER·11 min read
Build Your Own BPE Tokenizer — Learning Merge Rules, and Getting Punished by Japanese
Write the BPE trainer and encoder yourself. Why the artifact of training is an ordered rulebook rather than a vocabulary, how to stop recounting the corpus on every merge, why the first few thousand merge slots in Japanese are spent assembling characters, and how to run a vocabulary-size sweep that actually means something.
10
·How Transformers Work·★ MEMBER·PAPER·12 min read
Build Your Own Mini GPT — A Language Model in 300 Lines
Write a character-level GPT in PyTorch from an empty file: tokenizer, causally masked self-attention, training loop, and temperature sampling — then watch Shakespeare's formatting emerge from nothing but next-character prediction.
11
·How Transformers Work·★ MEMBER·PAPER·16 min read
Paper Walkthrough: Designing Qwen3.8-Next — Accuracy, Efficiency and Stability as One Problem
A ground-up read of the Qwen3.8-Flash-Next design report: the GDN hybrid, Qwen Sparse Attention, the Gated Residual and n-gram embeddings, judged the way the paper judges them — on loss, on cost, and on stability.
12
·How Transformers Work·★ MEMBER·PAPER·10 min read
Paper Explained: Why Gated DeltaNet Survives 4-Bit Quantization — NVFP4 W4A4 in a Hybrid 27B
A walkthrough of the paper that tested — and overturned — the belief that the recurrent half of a hybrid LLM is too fragile for 4-bit quantization, and explained mechanistically why log-space gates and the delta rule erase quantization noise.
13
·How Transformers Work·★ MEMBER·PAPER·10 min read
Paper Walkthrough: Stop Anchoring to Frame One — Scal3R's Multi-Reference Relative Pose Query
Long videos make online 3D reconstruction collapse. This paper traces the cause to extrapolation against a fixed first-frame anchor and fixes it by adding roughly 1% of trainable tokens to a frozen backbone. Explained from zero.