adafactor

Definition

An optimiser that stores its second-moment estimate in factored form — row and column statistics instead of a full per-parameter array — giving sublinear memory in the parameter count, per Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. That saving is why it has been used to train large Transformers where optimiser state rivals the model itself. The tradeoff is extra sensitivity to its schedule settings.