adam

Definition

An optimiser that keeps exponential moving averages of the gradient and of its square, using them to give each parameter its own effective step size. Introduced in Adam: A Method for Stochastic Optimization by Kingma and Ba, it became the default largely because it works acceptably with little tuning. In practice today you usually want AdamW, which decouples weight decay from the adaptive scaling.

Explainers using this term

• Beyond SGD — Adam, Second-Order Methods, and Constrained OptimizationAdam: A Method for Stochastic Optimization