adam

Definition

An optimiser that keeps exponential moving averages of the gradient and of its square, using them to give each parameter its own effective step size. Introduced in Adam: A Method for Stochastic Optimization by Kingma and Ba, it became the default largely because it works acceptably with little tuning. In practice today you usually want AdamW, which decouples weight decay from the adaptive scaling.

Explainers using this term

• Adam from Scratch — Two Moving Averages and the Correction Nobody ExplainsAdam: A Method for Stochastic Optimization
• Beyond SGD — Adam, Second-Order Methods, and Constrained OptimizationAdam: A Method for Stochastic Optimization