#memory-bandwidth

2 articles

01 ·How Transformers Work·FREE·PAPER·10 min read FlashAttention from Scratch — The Paradox of Doing More Math to Go Faster FlashAttention deliberately recomputes the same math in the backward pass. It does strictly more arithmetic and still runs faster — because on a GPU, moving numbers costs more than multiplying them. From counting HBM round trips, through the softmax wall that blocks tiling, to the online-softmax recurrence that breaks it, and the silent backend fallback that bites people in production. 02 ·Complexity·★ MEMBER·8 min read When Big-O and Your Benchmarks Disagree — Caches, Branches, and Memory Bandwidth Two O(n) programs can differ by orders of magnitude in the real world. This article unpacks what Big-O deliberately throws away — cache hierarchies, branch prediction, and memory bandwidth — and how to reason about each.