memory-efficient

Definition

Doing the same computation with a smaller memory footprint. Memory, not FLOPs, is usually the binding constraint on what model size, batch size, and context length fit on a given accelerator, which is why gradient checkpointing, quantization, activation recomputation, and KV-cache compression all fly this flag. Most of these trade memory for extra compute, so memory-efficient does not imply faster — sometimes the opposite.

Explainers using this term

• FlashAttention from Scratch — The Paradox of Doing More Math to Go FasterFlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• The GPU Memory Hierarchy — HBM, SRAM, Registers, and Why Movement WinsFlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness