preconditioned

Definition

Describes an update where the raw gradient is first multiplied by a matrix that rescales and rotates it. The point is ill-conditioning: when the loss stretches far more along some directions than others, plain gradient descent zig-zags, and preconditioning amounts to changing coordinates so the problem looks isotropic. Which matrix you use is precisely what distinguishes Adam, K-FAC and Shampoo from one another.