block-wise

Definition

Splitting a tensor into fixed-size blocks and treating each independently, typically with its own scaling constant. For quantization this matters because a single outlier only distorts the block it sits in, instead of stretching the scale for the entire tensor as a per-tensor scheme would. It is the trick behind 8-bit optimizers, which shrink optimizer state without destabilizing training.