flashattention

Definition

An IO-aware implementation of attention that tiles the computation into blocks kept in the GPU's fast on-chip SRAM, so the full attention matrix is never written to high-bandwidth memory. It is exact — the output matches standard attention rather than approximating it — while reducing memory from quadratic to linear in sequence length and cutting wall-clock time. It is a large part of why long context windows became practical, and it now ships inside the major training and inference stacks.

Explainers using this term

• FlashAttention from Scratch — The Paradox of Doing More Math to Go FasterFlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• The GPU Memory Hierarchy — HBM, SRAM, Registers, and Why Movement WinsFlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness