Learn › AI

How Transformers Work

Attention, positional encoding, and the architecture dissected

01 ·How Transformers Work·FREE·8 min read Attention from Scratch — The Heart of the Transformer, Explained Visually Self-attention, the core mechanism behind ChatGPT, explained from zero: analogy, intuition, matrix mechanics, and runnable numpy code. No math background required. 02 ·How Transformers Work·FREE·PAPER·10 min read Positional Encoding from Scratch — From Absolute Positions to RoPE A bare Transformer has no idea what word order is. Starting from why position information is needed at all, this article walks through sinusoidal absolute encodings, learned embeddings, and RoPE — the modern LLM standard — showing exactly why rotation encodes relative position. 03 ·How Transformers Work·FREE·PAPER·10 min read Tokenizers from Scratch — The Unit an LLM Cuts the World Into An LLM reads neither characters nor words. How BPE builds a vocabulary, what SentencePiece actually fixed, why some languages pay more for the same sentence, and what you trade away when you grow the vocabulary — worked by hand and in code, from zero. 04 ·How Transformers Work·FREE·PAPER·9 min read A Field Guide to Attention Variants — MQA, GQA, Sliding Windows, Linear Attention MQA, GQA, sliding windows and linear attention are not four unrelated tricks. One multiplication decides how large a KV cache gets, and every variant is a decision about which factor in it to attack. Lined up against that formula, the family tree shows exactly what each one gave up and what it bought. 05 ·How Transformers Work·FREE·PAPER·10 min read The Transformer, End to End — One Token's Journey from Embedding to Output Between typing a prompt and getting a word back, a single token gets handed a vector, rewritten in room after room, and finally turned back into language. Tokenizing, embedding, position, attention, feed-forward, residuals, and the output head — walked as one continuous trip rather than a pile of parts. 06 ·How Transformers Work·FREE·PAPER·10 min read FlashAttention from Scratch — The Paradox of Doing More Math to Go Faster FlashAttention deliberately recomputes the same math in the backward pass. It does strictly more arithmetic and still runs faster — because on a GPU, moving numbers costs more than multiplying them. From counting HBM round trips, through the softmax wall that blocks tiling, to the online-softmax recurrence that breaks it, and the silent backend fallback that bites people in production. 07 ·How Transformers Work·★ MEMBER·PAPER·11 min read How Long-Context LLMs Work — From RoPE Interpolation to Ring Attention A "128K context window" is two different walls, knocked down by two unrelated families of tricks. This walks through positional interpolation, NTK-aware scaling and YaRN for the position wall; sliding windows and ring attention for the compute wall; and how to read a needle-in-a-haystack chart without being fooled by it. 08 ·How Transformers Work·★ MEMBER·PAPER·9 min read Encoder or Decoder — The Fork in the Road Between BERT and GPT One masked triangle in the attention table is what separated BERT from GPT. This piece works through bidirectional versus autoregressive with the equations and an interactive figure, then asks why generation won and where encoders are still the first choice. 09 ·How Transformers Work·★ MEMBER·PAPER·11 min read Build Your Own BPE Tokenizer — Learning Merge Rules, and Getting Punished by Japanese Write the BPE trainer and encoder yourself. Why the artifact of training is an ordered rulebook rather than a vocabulary, how to stop recounting the corpus on every merge, why the first few thousand merge slots in Japanese are spent assembling characters, and how to run a vocabulary-size sweep that actually means something. 10 ·How Transformers Work·★ MEMBER·PAPER·12 min read Build Your Own Mini GPT — A Language Model in 300 Lines Write a character-level GPT in PyTorch from an empty file: tokenizer, causally masked self-attention, training loop, and temperature sampling — then watch Shakespeare's formatting emerge from nothing but next-character prediction. 11 ·How Transformers Work·★ MEMBER·PAPER·16 min read Qwen3.8-Next design report explained (arXiv:2608.30320) A ground-up read of the Qwen3.8-Flash-Next report: 125B total parameters with 6B activated, the GDN hybrid, Qwen Sparse Attention (7.6x faster prefill at 1M context), the Gated Residual and n-gram embeddings — judged on loss, cost and training stability. 12 ·How Transformers Work·★ MEMBER·PAPER·11 min read Paper Explained: Why Gated DeltaNet Survives 4-Bit Quantization — NVFP4 W4A4 in a Hybrid 27B A walkthrough of the paper that tested — and overturned — the belief that the recurrent half of a hybrid LLM is too fragile for 4-bit quantization, and explained mechanistically why log-space gates and the delta rule erase quantization noise. 13 ·How Transformers Work·★ MEMBER·PAPER·11 min read Paper Walkthrough: Stop Anchoring to Frame One — Scal3R's Multi-Reference Relative Pose Query Long videos make online 3D reconstruction collapse. This paper traces the cause to extrapolation against a fixed first-frame anchor and fixes it by adding roughly 1% of trainable tokens to a frozen backbone. Explained from zero. 14 ·How Transformers Work·★ MEMBER·PAPER·15 min read Paper Walkthrough: SAS — Training a Sparse-Attention Block Selector End-to-End With the Language Modeling Loss The selector that decides which context blocks a query reads has traditionally been trained by imitating the dense model's attention distribution. SAS injects the selector's scores into the softmax as log-space gates, so the gradient that Top-K used to block flows straight from next-token prediction into the selector. We follow the paper through its four-axis ablation, the gradient formulas behind each choice, and the 10.6–15.5 point GPQA-Diamond gain at a 1024-token budget. 15 ·How Transformers Work·★ MEMBER·PAPER·7 min read Paper Explained: LimiX-2 — A Tabular Foundation Model That Learns the Whole Table, Not Just the Answer Column LimiX-2 from the ground up: the shift from PFNs that predict a single target to Contextual Mechanism Networks that learn a table's joint distribution, cell-level representations, pretraining on synthetic causal data, and what the three benchmarks do and don't show. 16 ·How Transformers Work·★ MEMBER·PAPER·9 min read Paper Deep Dive: Your Transformer Can Hold Two Thoughts at Once — Blend Two Texts and the LLM Still Remembers Both Average the embeddings of two unrelated texts, run a single forward pass, and an LLM's output still keeps both streams' next-token candidates near the top. We walk through the paper's evidence that this 'superposition linearity' fades during pretraining, comes back with light fine-tuning, and can be partially decoded into two continuations.