#momentum

2 articles

01 ·Training & Alignment·FREE·PAPER·18 min read Adam from Scratch — Two Moving Averages and the Correction Nobody Explains Adam is the default optimizer in almost every deep-learning codebase, and almost nobody can say what its four lines do. This article assembles it from the two problems it solves — the zigzag of plain SGD and the mismatched gradient scales between layers — then takes the bias correction apart with numbers: without it, Adam's very first step is 3.2× too large and the error peaks at 6.6×. Ends on the traps: weight decay belongs in AdamW, not Adam, and the epsilon sits outside the square root for a reason. 02 ·Calculus & Optimization·★ MEMBER·PAPER·12 min read Beyond SGD — Adam, Second-Order Methods, and Constrained Optimization What exactly is momentum accumulating? What does each of Adam's four lines do? What did AdamW fix? And why does nobody train an LLM with second-order methods that are supposedly faster? Metaphor, equations, live figures, code, and production practice — no prerequisites assumed.