shampoo

Definition

A preconditioned optimiser that maintains a separate preconditioning matrix for each dimension of a parameter tensor, from Shampoo: Preconditioned Stochastic Tensor Optimization. Compared with Adam's purely diagonal scaling it captures correlations between coordinates, at the cost of computing inverse matrix roots. Interest has revived through implementations that amortise that cost across steps for large-scale training.