blockwise

Definition

Processing a sequence or matrix in tiles rather than all at once. Standard attention materialises an N×N score matrix; blockwise attention computes it tile by tile and accumulates the softmax statistics incrementally, so peak memory stops scaling with the square of the sequence length. Both FlashAttention and Ring Attention are built on this, which is why long-context work is full of the word.