io-awareness

Definition

Designing an algorithm around data movement between levels of the memory hierarchy rather than around arithmetic operations. On modern accelerators, floating-point throughput has outrun memory bandwidth, so the number of reads and writes to high-bandwidth memory often determines runtime while the FLOP count does not. Popularized as the stated design principle of FlashAttention, IO-awareness is now the default lens for writing fast kernels.

Explainers using this term

• FlashAttention from Scratch — The Paradox of Doing More Math to Go FasterFlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• The GPU Memory Hierarchy — HBM, SRAM, Registers, and Why Movement WinsFlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness