#rmsprop

1 articles

01 ·Training & Alignment·FREE·PAPER·18 min read Adam from Scratch — Two Moving Averages and the Correction Nobody Explains Adam is the default optimizer in almost every deep-learning codebase, and almost nobody can say what its four lines do. This article assembles it from the two problems it solves — the zigzag of plain SGD and the mismatched gradient scales between layers — then takes the bias correction apart with numbers: without it, Adam's very first step is 3.2× too large and the error peaks at 6.6×. Ends on the traps: weight decay belongs in AdamW, not Adam, and the epsilon sits outside the square root for a reason.