#flash-attention
2 articles
01
·How Transformers Work·★ MEMBER·PAPER·10 min read
FlashAttention from Scratch — The Paradox of Doing More Math to Go Faster
FlashAttention deliberately recomputes the same math in the backward pass. It does strictly more arithmetic and still runs faster — because on a GPU, moving numbers costs more than multiplying them. From counting HBM round trips, through the softmax wall that blocks tiling, to the online-softmax recurrence that breaks it, and the silent backend fallback that bites people in production.
02
·Computer Architecture·★ MEMBER·PAPER·9 min read
The GPU Memory Hierarchy — HBM, SRAM, Registers, and Why Movement Wins
What sets a GPU's speed is not the arithmetic units but where the data sits — registers, shared memory, L2 or HBM — and how many times it is moved. Capacities and bandwidths by order of magnitude, arithmetic intensity and tiling, a roofline per level of the hierarchy, and finally FlashAttention: more FLOPs, less time.