Students Professional
JA EN

#transformer

22 articles

01 ·How Transformers Work·★ MEMBER·PAPER·10 min read Paper Walkthrough: Stop Anchoring to Frame One — Scal3R's Multi-Reference Relative Pose Query Long videos make online 3D reconstruction collapse. This paper traces the cause to extrapolation against a fixed first-frame anchor and fixes it by adding roughly 1% of trainable tokens to a frozen backbone. Explained from zero. 02 ·★ MEMBER·PAPER·14 min read Paper Walkthrough: Eleven Frames Are Enough — Revisiting Local Context for Long-Horizon Streaming 3D Reconstruction How do you track a camera and rebuild a scene from a video that never ends? ABot-Recon throws away long-range memory entirely and bets on the last twelve frames. A ground-up walkthrough of the paper. 03 ·How Transformers Work·★ MEMBER·PAPER·16 min read Paper Walkthrough: Designing Qwen3.8-Next — Accuracy, Efficiency and Stability as One Problem A ground-up read of the Qwen3.8-Flash-Next design report: the GDN hybrid, Qwen Sparse Attention, the Gated Residual and n-gram embeddings, judged the way the paper judges them — on loss, on cost, and on stability. 04 ·How Transformers Work·★ MEMBER·PAPER·11 min read How Long-Context LLMs Work — From RoPE Interpolation to Ring Attention A "128K context window" is two different walls, knocked down by two unrelated families of tricks. This walks through positional interpolation, NTK-aware scaling and YaRN for the position wall; sliding windows and ring attention for the compute wall; and how to read a needle-in-a-haystack chart without being fooled by it. 05 ·How Transformers Work·★ MEMBER·PAPER·10 min read FlashAttention from Scratch — The Paradox of Doing More Math to Go Faster FlashAttention deliberately recomputes the same math in the backward pass. It does strictly more arithmetic and still runs faster — because on a GPU, moving numbers costs more than multiplying them. From counting HBM round trips, through the softmax wall that blocks tiling, to the online-softmax recurrence that breaks it, and the silent backend fallback that bites people in production. 06 ·How Transformers Work·★ MEMBER·PAPER·11 min read Build Your Own BPE Tokenizer — Learning Merge Rules, and Getting Punished by Japanese Write the BPE trainer and encoder yourself. Why the artifact of training is an ordered rulebook rather than a vocabulary, how to stop recounting the corpus on every merge, why the first few thousand merge slots in Japanese are spent assembling characters, and how to run a vocabulary-size sweep that actually means something. 07 ·How Transformers Work·★ MEMBER·PAPER·12 min read Build Your Own Mini GPT — A Language Model in 300 Lines Write a character-level GPT in PyTorch from an empty file: tokenizer, causally masked self-attention, training loop, and temperature sampling — then watch Shakespeare's formatting emerge from nothing but next-character prediction. 08 ·How Transformers Work·★ MEMBER·PAPER·9 min read Encoder or Decoder — The Fork in the Road Between BERT and GPT One masked triangle in the attention table is what separated BERT from GPT. This piece works through bidirectional versus autoregressive with the equations and an interactive figure, then asks why generation won and where encoders are still the first choice. 09 ·How Transformers Work·FREE·PAPER·10 min read The Transformer, End to End — One Token's Journey from Embedding to Output Between typing a prompt and getting a word back, a single token gets handed a vector, rewritten in room after room, and finally turned back into language. Tokenizing, embedding, position, attention, feed-forward, residuals, and the output head — walked as one continuous trip rather than a pile of parts. 10 ·Model Families·★ MEMBER·9 min read The GPT Lineage — Design Thinking from GPT-1 to Today A generation-by-generation walk from GPT-1 to GPT-4o and the reasoning models, asking what changed and what deliberately did not. At the centre sits one machine that only ever predicts the next token; what moved was scale, the order of training, and how the output was disciplined. 11 ·How Transformers Work·FREE·PAPER·10 min read Tokenizers from Scratch — The Unit an LLM Cuts the World Into An LLM reads neither characters nor words. How BPE builds a vocabulary, what SentencePiece actually fixed, why some languages pay more for the same sentence, and what you trade away when you grow the vocabulary — worked by hand and in code, from zero. 12 ·Time Series·★ MEMBER·PAPER·11 min read Do Transformers Actually Work on Time Series? — The Argument and the Practical Answer In 2022 a single linear layer beat the whole crop of time-series Transformers on the standard benchmarks. The culprit turned out to be tokenization, not attention — a diagnosis PatchTST fixed by patching and iTransformer by transposing the axes. And yet gradient boosting keeps winning the practitioner competitions, for reasons that come down to the shape of real data. 13 ·How Transformers Work·★ MEMBER·PAPER·9 min read A Field Guide to Attention Variants — MQA, GQA, Sliding Windows, Linear Attention MQA, GQA, sliding windows and linear attention are not four unrelated tricks. One multiplication decides how large a KV cache gets, and every variant is a decision about which factor in it to attack. Lined up against that formula, the family tree shows exactly what each one gave up and what it bought. 14 ·CNNs & Image Recognition·★ MEMBER·PAPER·9 min read Object Detection from Scratch (from YOLO to DETR) A from-zero guide to object detection: the classic toolkit of two-stage detectors, anchors, and NMS — and how DETR reframed the whole problem to make all of it unnecessary. Primary source: the DETR paper. 15 ·Agents·★ MEMBER·PAPER·8 min read Paper Walkthrough: Metis — A 'Memory Foundation Model' That Moves Agent Memory Inside the Model Agent memory today is mostly bolted on from the outside via RAG. This paper proposes memory foundation models — models whose forward pass natively stores, forgets, and updates information — and builds Metis, the first prototype. A from-scratch walkthrough of how it works, how well it works, and where it breaks. 16 ·Paper Deep-Dives·★ MEMBER·PAPER·9 min read Mixture of Experts (MoE) from Scratch — Routing and Load Balancing in the Switch Transformer A ground-up explanation of Mixture of Experts, the sparse architecture behind today's largest LLMs, built strictly from the Switch Transformer paper: the router math, expert capacity, the load-balancing loss, and the three tricks that make sparse training stable. 17 ·VLMs & Multimodal·★ MEMBER·PAPER·8 min read BEV Representations From Scratch — Fusing Multiple Cameras Into One Top-Down Map How a self-driving car turns six camera feeds into a single top-down map. Starting from perspective projection, we build up to the two big design philosophies: LSS, which pushes features into 3D via a predicted depth distribution, and Transformer-style methods like BEVFormer that pull information with BEV queries. 18 ·VLMs & Multimodal·★ MEMBER·PAPER·7 min read Paper Deep Dive — ViT: Treating an Image Like a Sentence A reading of the ViT paper (Dosovitskiy et al., 2020/2021) grounded strictly in its own text: the move of treating 16x16 patches as words, what the position-embedding ablation actually showed, the price of dropping the convolutional inductive bias, and how conditional the claim 'beats CNNs at scale' really is. 19 ·How Transformers Work·FREE·PAPER·10 min read Positional Encoding from Scratch — From Absolute Positions to RoPE A bare Transformer has no idea what word order is. Starting from why position information is needed at all, this article walks through sinusoidal absolute encodings, learned embeddings, and RoPE — the modern LLM standard — showing exactly why rotation encodes relative position. 20 ·Inference & Serving·FREE·7 min read The KV Cache from Scratch — The Heart of Fast Inference An LLM emits one token at a time. Written naively, every single token costs a full recomputation of the whole sequence — a spectacular waste. The keys and values of past tokens never change again, and that one fact drops an entire order of magnitude. What you pay instead is memory, in an amount you can work out yourself, and that is why batch size and context length hit a ceiling. 21 ·Paper Deep-Dives·★ MEMBER·PAPER·12 min read Paper Deep Dive — Attention Is All You Need: What Dropping Recurrence Actually Proved A close reading of the Transformer paper grounded strictly in its own text: the scaled dot-product equation, why the square root of d_k is there, what the ablations exposed, and the limits the authors themselves flagged. 22 ·How Transformers Work·FREE·8 min read Attention from Scratch — The Heart of the Transformer, Explained Visually Self-attention, the core mechanism behind ChatGPT, explained from zero: analogy, intuition, matrix mechanics, and runnable numpy code. No math background required.