Students Professional
JA EN

#attention

13 articles

01 ·Time Series·★ MEMBER·PAPER·12 min read H3-World, Explained — Turning Language Understanding into World Control How to turn a 33B video generator into a controllable world model without bolting on an action module: express actions as sentences, bind each sentence to a video-latent interval, and use attention routing to stop the crosstalk. Only 0.199% of parameters are trained. 02 ·How Transformers Work·★ MEMBER·PAPER·11 min read How Long-Context LLMs Work — From RoPE Interpolation to Ring Attention A "128K context window" is two different walls, knocked down by two unrelated families of tricks. This walks through positional interpolation, NTK-aware scaling and YaRN for the position wall; sliding windows and ring attention for the compute wall; and how to read a needle-in-a-haystack chart without being fooled by it. 03 ·How Transformers Work·★ MEMBER·PAPER·10 min read FlashAttention from Scratch — The Paradox of Doing More Math to Go Faster FlashAttention deliberately recomputes the same math in the backward pass. It does strictly more arithmetic and still runs faster — because on a GPU, moving numbers costs more than multiplying them. From counting HBM round trips, through the softmax wall that blocks tiling, to the online-softmax recurrence that breaks it, and the silent backend fallback that bites people in production. 04 ·How Transformers Work·FREE·PAPER·10 min read The Transformer, End to End — One Token's Journey from Embedding to Output Between typing a prompt and getting a word back, a single token gets handed a vector, rewritten in room after room, and finally turned back into language. Tokenizing, embedding, position, attention, feed-forward, residuals, and the output head — walked as one continuous trip rather than a pile of parts. 05 ·How Transformers Work·FREE·PAPER·10 min read The Transformer, End to End — One Token's Journey from Embedding to Output Between typing a prompt and getting a word back, a single token gets handed a vector, rewritten in room after room, and finally turned back into language. Tokenizing, embedding, position, attention, feed-forward, residuals, and the output head — walked as one continuous trip rather than a pile of parts. 06 ·How Transformers Work·★ MEMBER·PAPER·9 min read A Field Guide to Attention Variants — MQA, GQA, Sliding Windows, Linear Attention MQA, GQA, sliding windows and linear attention are not four unrelated tricks. One multiplication decides how large a KV cache gets, and every variant is a decision about which factor in it to attack. Lined up against that formula, the family tree shows exactly what each one gave up and what it bought. 07 ·How Transformers Work·★ MEMBER·PAPER·9 min read A Field Guide to Attention Variants — MQA, GQA, Sliding Windows, Linear Attention MQA, GQA, sliding windows and linear attention are not four unrelated tricks. One multiplication decides how large a KV cache gets, and every variant is a decision about which factor in it to attack. Lined up against that formula, the family tree shows exactly what each one gave up and what it bought. 08 ·How Transformers Work·FREE·PAPER·10 min read Positional Encoding from Scratch — From Absolute Positions to RoPE A bare Transformer has no idea what word order is. Starting from why position information is needed at all, this article walks through sinusoidal absolute encodings, learned embeddings, and RoPE — the modern LLM standard — showing exactly why rotation encodes relative position. 09 ·How Transformers Work·FREE·PAPER·10 min read Positional Encoding from Scratch — From Absolute Positions to RoPE A bare Transformer has no idea what word order is. Starting from why position information is needed at all, this article walks through sinusoidal absolute encodings, learned embeddings, and RoPE — the modern LLM standard — showing exactly why rotation encodes relative position. 10 ·Paper Deep-Dives·★ MEMBER·PAPER·12 min read Paper Deep Dive — Attention Is All You Need: What Dropping Recurrence Actually Proved A close reading of the Transformer paper grounded strictly in its own text: the scaled dot-product equation, why the square root of d_k is there, what the ablations exposed, and the limits the authors themselves flagged. 11 ·Paper Deep-Dives·★ MEMBER·PAPER·12 min read Paper Deep Dive — Attention Is All You Need: What Dropping Recurrence Actually Proved A close reading of the Transformer paper grounded strictly in its own text: the scaled dot-product equation, why the square root of d_k is there, what the ablations exposed, and the limits the authors themselves flagged. 12 ·How Transformers Work·FREE·8 min read Attention from Scratch — The Heart of the Transformer, Explained Visually Self-attention, the core mechanism behind ChatGPT, explained from zero: analogy, intuition, matrix mechanics, and runnable numpy code. No math background required. 13 ·How Transformers Work·FREE·8 min read Attention from Scratch — The Heart of the Transformer, Explained Visually Self-attention, the core mechanism behind ChatGPT, explained from zero: analogy, intuition, matrix mechanics, and runnable numpy code. No math background required.