Calculus & Optimization
Partial derivatives, gradients, the chain rule, convexity, Lagrange
01
·Calculus & Optimization·FREE·7 min read
Calculus for AI — The Gradient Is an Arrow Saying Which Way Is Better
No epsilon-delta limits, no integration by parts. Training is measuring a slope and stepping the other way. A derivative is a multiplier, a gradient is a list of slopes, the chain rule is multiplication — and Jacobians and Hessians only need to be recognised, not computed.
02
·Calculus & Optimization·★ MEMBER·PAPER·9 min read
Convexity and Optimization — Why Deep Learning Works Even Though It Isn't Convex
Optimization textbooks teach a stark divide: convex problems are solvable, non-convex ones come with no guarantees. So why does deep learning — whose loss surface is provably non-convex — work at all? From convex sets and functions to saddle points and flat minima, this article connects the whole story in the language of landscapes.
03
·Calculus & Optimization·★ MEMBER·PAPER·12 min read
Beyond SGD — Adam, Second-Order Methods, and Constrained Optimization
What exactly is momentum accumulating? What does each of Adam's four lines do? What did AdamW fix? And why does nobody train an LLM with second-order methods that are supposedly faster? Metaphor, equations, live figures, code, and production practice — no prerequisites assumed.
04
·Calculus & Optimization·★ MEMBER·11 min read
Matrix Calculus from Scratch — Derive the Backward Pass Yourself
Where does the transpose in ∂L/∂W = XᵀG actually come from? Matrix calculus is not a formula sheet to memorize — it is one move: rotate dX to the right inside a trace. From denominator layout and shape-checking to the gradients of a linear layer and softmax + cross-entropy, ending with a double-precision gradient check.
05
·Calculus & Optimization·★ MEMBER·11 min read
Jacobians and Hessians — Multivariable Calculus, Drawn
The Jacobian is a magnifying glass at a point; the Hessian is how sharply the ground curves there. From local linearization to what eigenvalues say about the terrain, why Newton's method is fast on paper but absent in practice, and how to get Hessian information in a billion dimensions without ever building the matrix.
06
·Calculus & Optimization·★ MEMBER·13 min read
Calculus of Variations — What It Means to Differentiate a Function
When the thing you are searching for is a shape rather than a number, differentiation has to be reinvented. Functionals and the first variation, the Euler–Lagrange equation, why the fastest slide is a cycloid, and how all of it lands on the ELBO — the variational problem machine learning runs most.