#attention

16 articles

01 ·Inference & Serving·★ MEMBER·PAPER·10 min read Paper Explained: FreeFlow — Optical Flow with a Hierarchical Transformer and No Flow-Specific Parts FreeFlow drops correlation volumes, warping, and iterative refinement, and reaches the top of Sintel, KITTI-2015, and Spring using only window, shifted-window, and low-resolution global attention. We walk through it from zero. 02 ·How Transformers Work·★ MEMBER·PAPER·15 min read Paper Walkthrough: SAS — Training a Sparse-Attention Block Selector End-to-End With the Language Modeling Loss The selector that decides which context blocks a query reads has traditionally been trained by imitating the dense model's attention distribution. SAS injects the selector's scores into the softmax as log-space gates, so the gradient that Top-K used to block flows straight from next-token prediction into the selector. We follow the paper through its four-axis ablation, the gradient formulas behind each choice, and the 10.6–15.5 point GPQA-Diamond gain at a 1024-token budget. 03 ·Inference & Serving·★ MEMBER·PAPER·12 min read Paper walkthrough: BeaconKV — evicting KV cache by predicting when a reasoning model will re-read its own plan Deep into a long chain of thought, a reasoning model occasionally looks back at the plan it wrote at the start. The queries that do this cluster into a handful of groups in embedding space — so keeping a few dozen cluster representatives (beacons) is enough to predict which KV pairs will be revisited. A training-free method with up to 5.8× less memory and over 4.3× more throughput, read through its equations and code. 04 ·Time Series·★ MEMBER·PAPER·12 min read H3-World: Turning Language Understanding into World Control — Paper Explained H3-World turns a 33B video generator into a controllable world model without bolting on an action module: express actions as sentences, bind each sentence to a video-latent interval, and use attention routing to stop the crosstalk. Only 0.199% of parameters are trained. 05 ·How Transformers Work·★ MEMBER·PAPER·11 min read How Long-Context LLMs Work — From RoPE Interpolation to Ring Attention A "128K context window" is two different walls, knocked down by two unrelated families of tricks. This walks through positional interpolation, NTK-aware scaling and YaRN for the position wall; sliding windows and ring attention for the compute wall; and how to read a needle-in-a-haystack chart without being fooled by it. 06 ·How Transformers Work·FREE·PAPER·10 min read FlashAttention from Scratch — The Paradox of Doing More Math to Go Faster FlashAttention deliberately recomputes the same math in the backward pass. It does strictly more arithmetic and still runs faster — because on a GPU, moving numbers costs more than multiplying them. From counting HBM round trips, through the softmax wall that blocks tiling, to the online-softmax recurrence that breaks it, and the silent backend fallback that bites people in production. 07 ·How Transformers Work·FREE·PAPER·10 min read The Transformer, End to End — One Token's Journey from Embedding to Output Between typing a prompt and getting a word back, a single token gets handed a vector, rewritten in room after room, and finally turned back into language. Tokenizing, embedding, position, attention, feed-forward, residuals, and the output head — walked as one continuous trip rather than a pile of parts. 08 ·How Transformers Work·FREE·PAPER·10 min read The Transformer, End to End — One Token's Journey from Embedding to Output Between typing a prompt and getting a word back, a single token gets handed a vector, rewritten in room after room, and finally turned back into language. Tokenizing, embedding, position, attention, feed-forward, residuals, and the output head — walked as one continuous trip rather than a pile of parts. 09 ·How Transformers Work·FREE·PAPER·9 min read A Field Guide to Attention Variants — MQA, GQA, Sliding Windows, Linear Attention MQA, GQA, sliding windows and linear attention are not four unrelated tricks. One multiplication decides how large a KV cache gets, and every variant is a decision about which factor in it to attack. Lined up against that formula, the family tree shows exactly what each one gave up and what it bought. 10 ·How Transformers Work·FREE·PAPER·9 min read A Field Guide to Attention Variants — MQA, GQA, Sliding Windows, Linear Attention MQA, GQA, sliding windows and linear attention are not four unrelated tricks. One multiplication decides how large a KV cache gets, and every variant is a decision about which factor in it to attack. Lined up against that formula, the family tree shows exactly what each one gave up and what it bought. 11 ·How Transformers Work·FREE·PAPER·10 min read Positional Encoding from Scratch — From Absolute Positions to RoPE A bare Transformer has no idea what word order is. Starting from why position information is needed at all, this article walks through sinusoidal absolute encodings, learned embeddings, and RoPE — the modern LLM standard — showing exactly why rotation encodes relative position. 12 ·How Transformers Work·FREE·PAPER·10 min read Positional Encoding from Scratch — From Absolute Positions to RoPE A bare Transformer has no idea what word order is. Starting from why position information is needed at all, this article walks through sinusoidal absolute encodings, learned embeddings, and RoPE — the modern LLM standard — showing exactly why rotation encodes relative position. 13 ·Paper Deep-Dives·FREE·PAPER·12 min read Paper Deep Dive — Attention Is All You Need: What Dropping Recurrence Actually Proved A close reading of the Transformer paper grounded strictly in its own text: the scaled dot-product equation, why the square root of d_k is there, what the ablations exposed, and the limits the authors themselves flagged. 14 ·Paper Deep-Dives·FREE·PAPER·12 min read Paper Deep Dive — Attention Is All You Need: What Dropping Recurrence Actually Proved A close reading of the Transformer paper grounded strictly in its own text: the scaled dot-product equation, why the square root of d_k is there, what the ablations exposed, and the limits the authors themselves flagged. 15 ·How Transformers Work·FREE·8 min read Attention from Scratch — The Heart of the Transformer, Explained Visually Self-attention, the core mechanism behind ChatGPT, explained from zero: analogy, intuition, matrix mechanics, and runnable numpy code. No math background required. 16 ·How Transformers Work·FREE·8 min read Attention from Scratch — The Heart of the Transformer, Explained Visually Self-attention, the core mechanism behind ChatGPT, explained from zero: analogy, intuition, matrix mechanics, and runnable numpy code. No math background required.