#gradient-clipping
2 articles
01
·Training & Alignment·FREE·PAPER·11 min read
Diagnosing Broken Training — Telling Divergence, NaN, and Plateaus Apart
Training breaks in exactly three ways: it diverges, it goes NaN, or it stalls. Built around a symptom-to-cause table, this article shows why divergence is a threshold effect (with the math and an interactive figure), how to pin down where a NaN was born, and how to isolate the cause of a plateau — assuming no prior knowledge.
02
·Calculus & Optimization·★ MEMBER·PAPER·12 min read
Beyond SGD — Adam, Second-Order Methods, and Constrained Optimization
What exactly is momentum accumulating? What does each of Adam's four lines do? What did AdamW fix? And why does nobody train an LLM with second-order methods that are supposedly faster? Metaphor, equations, live figures, code, and production practice — no prerequisites assumed.