#learning-rate

2 articles

01 ·Calculus & Optimization·FREE·PAPER·10 min read Gradient Descent from Scratch — The Geometry of Learning Every model that learns is walking downhill on a surface it cannot see all of. This article builds gradient descent from that one picture — the loss surface, the gradient as steepest ascent, the update rule w ← w − η∇L — then works out why the learning rate has a hard ceiling, why ravines make it zigzag, and what local minima really cost in high dimensions. 02 ·Training & Alignment·FREE·PAPER·10 min read Learning Rate Schedules — Why Warmup and Why Cosine The learning rate is not a fixed number — it is a curve you design across the whole run. Why we deliberately start slow (warmup), why we come down along a cosine, and what else has to move when batch size changes. Equations, live figures, PyTorch code, and the mistakes that actually break runs.