#second-order

1 articles

01 ·Calculus & Optimization·★ MEMBER·PAPER·12 min read Beyond SGD — Adam, Second-Order Methods, and Constrained Optimization What exactly is momentum accumulating? What does each of Adam's four lines do? What did AdamW fix? And why does nobody train an LLM with second-order methods that are supposedly faster? Metaphor, equations, live figures, code, and production practice — no prerequisites assumed.