Gradient Descent from Scratch — The Geometry of Learning

Every model that learns is walking downhill on a surface it cannot see all of. This article builds gradient descent from that one picture — the loss surface, the gradient as steepest ascent, the update rule w ← w − η∇L — then works out why the learning rate has a hard ceiling, why ravines make it zigzag, and what local minima really cost in high dimensions.

An overview of gradient descent optimization algorithms

Learning is walking downhill in fog

Almost every model that "learns" is doing one thing underneath: adjusting its parameters to make a mistake number smaller. That mistake number is the loss, written $L$. It takes in the model's parameters $w$ (there may be billions of them) and returns a single number saying how badly the model is doing right now.

Picture the loss as a landscape. Every possible setting of the parameters is a spot on the ground, and the height at that spot is the loss there. Training is the search for the lowest valley. The catch is that you cannot see the whole landscape — with a billion parameters, nobody can. All you can feel is the slope directly under your feet. Gradient descent is the strategy of a hiker lost in fog: feel which way is downhill, take a step, feel again, repeat. This article builds that idea up in the usual order — picture, intuition, the update rule taken apart term by term, code — and then follows it into the two things that actually go wrong in practice: a learning rate set too high, and a valley shaped like a ravine.

The gradient: the compass that points straight uphill

The tool that tells you the slope is the gradient, written $\nabla L$. For a function of many variables, the gradient is the list of its partial derivatives — one number per parameter, saying how much the loss changes if you nudge that single parameter and hold the rest still.

$$\nabla L(w) = \left[\frac{\partial L}{\partial w_1},\ \frac{\partial L}{\partial w_2},\ \dots,\ \frac{\partial L}{\partial w_n}\right]$$

In words, the gradient is just a list with one entry per parameter, and each entry tells you how fast the loss rises if you push only that one parameter and leave the rest alone. Stacked together, those entries form a vector that points toward the steepest climb.

Read as a sentence: assemble, for each parameter in turn, the answer to "if I push only this one knob, how fast does the loss rise?" — and stack those answers into a vector. The single most important fact about that vector is its direction. The gradient points in the direction of steepest ascent — the compass bearing along which the loss climbs fastest — and its length is how steep that climb is.

Why steepest ascent and not descent? The rate at which the loss changes as you move a tiny step in some unit direction $u$ is the dot product $\nabla L \cdot u$. A dot product is largest when the two vectors line up, so the fastest increase is in the direction of $\nabla L$ itself. To go down fastest, you therefore walk in exactly the opposite direction, $-\nabla L$. That minus sign is the entire idea of gradient descent in one character.

The update rule: one line, taken apart

Here is the rule the hiker follows, over and over:

In words, take your current parameters, compute the uphill compass at that spot, flip it to point downhill, scale it by the learning rate to set the step size, and move there. That moves you one step toward lower loss.

Take it one piece at a time. $w$ is the current parameters — where you are standing. $\nabla L(w)$ is the gradient there — the uphill compass. Putting a minus in front of it, $-\nabla L(w)$, flips it to point downhill. And $\eta$ (the Greek letter eta) is the learning rate: a small positive number that scales how big a step you take in that downhill direction. The arrow $\leftarrow$ means "compute the right-hand side, then overwrite $w$ with it." Do that, and you have moved one step toward lower loss.

That is the whole algorithm. Everything else in optimization — momentum, Adam, learning-rate schedules — is a refinement of this single line. Repeat it enough times and, on a well-behaved surface, you slide to the bottom of a valley, where the ground is flat, the gradient is zero, and the update stops moving you: $\nabla L = 0$ means $w - \eta \cdot 0 = w$, so you have arrived at a stationary point.

The learning rate has a hard ceiling

The learning rate $\eta$ is the single most consequential dial in all of training, and it fails in two opposite ways.

Set it too small and each step is a timid shuffle. You do head downhill, but it takes an enormous number of steps to get anywhere — training that could take an hour takes a week, and the process can stall on a nearly flat stretch long before it reaches the bottom.

Set it too large and you overshoot the valley floor on every step, landing further up the far wall than you started. The next step overshoots the other way, worse. The loss doesn't just fail to fall — it climbs, then explodes to infinity or NaN. This is not a vague risk; it has a precise threshold. On a simple bowl-shaped loss $L(w) = \tfrac{1}{2}a w^2$, the update becomes $w \leftarrow (1 - \eta a)\,w$, which shrinks $w$ toward zero only while $|1 - \eta a| < 1$ — that is, only while

$$\eta < \frac{2}{a}, \qquad a = L''(w).$$

In words, the largest learning rate you can safely use is set by the curvature: the sharper the bowl bends, the smaller that ceiling. Once $\eta$ crosses $2/a$, every step makes the loss worse.

The quantity $a = L''$ is the curvature — how sharply the bowl bends. So the ceiling on a usable learning rate is set by the curvature of the surface: the sharper the valley, the smaller the largest step you can take before you bounce out. Cross $2/a$ and every step makes things worse, guaranteed. This is why "the loss went to NaN" is, nine times out of ten, a learning rate that was too big.

Ravines: where a single learning rate can't win

The curvature above was a single number because the bowl was symmetric. Real loss surfaces are not. A common and painful shape is a ravine: a valley that is steeply walled in one direction and almost flat along its length, like a canyon floor.

That shape is a trap for plain gradient descent, because the gradient at any point on the canyon wall points mostly across the ravine (toward the near wall) rather than along it (toward the exit). So the hiker bounces back and forth between the two steep walls, making painfully slow progress down the gentle slope toward the actual minimum. The steep direction demands a small learning rate to avoid overshooting; the flat direction wants a large one to make progress. One dial cannot satisfy both, and you get the classic zigzag. (The ratio of the sharpest curvature to the flattest — the condition number — is exactly how bad the zigzag gets, and it is what preconditioners and second-order methods exist to fix; the sequel Beyond SGD picks up there.)

The standard patch is momentum: instead of stepping by the raw gradient each time, accumulate a running average of recent gradients and step by that. The side-to-side bounces cancel out because they keep reversing sign, while the consistent down-the-canyon component adds up — like a heavy ball that smooths out the rattling and keeps its downhill roll.

Local minima, saddle points, and why depth still works

A stationary point ($\nabla L = 0$) is not necessarily the bottom of the world. It might be a local minimum — the lowest point in its immediate neighborhood, but higher than some deeper valley elsewhere — or the single global minimum, the true lowest point. Gradient descent only ever feels the local slope, so in principle it can settle into a local minimum and stop, never knowing a better valley existed over the ridge.

For decades this was the standard fear about training neural networks. The modern understanding is more reassuring, and it is a story about high dimensions. In a space of millions of parameters, a stationary point is a local minimum only if the surface curves upward in every single one of those directions at once — and that is astronomically unlikely by chance. Far more common is the saddle point: a spot that curves up in some directions and down in others, flat at the center like a mountain pass or a horse's saddle. Saddles have an escape route — a downhill direction still exists — so with a little noise the optimizer slides off them and keeps going. Empirically, the many minima that deep networks do reach tend to be about equally good, which is a large part of why training enormous models works at all despite the loss being provably non-convex. The landscape view of that surprise is laid out in Convexity and Optimization.

Seeing it in code

The whole algorithm is a short loop. Here it is minimizing a plain quadratic bowl, $L(w) = w^2$, whose gradient is $\nabla L = 2w$:

import numpy as np

def loss(w):  return w**2
def grad(w):  return 2 * w        # dL/dw for L = w^2

w = 5.0          # start far from the minimum at w = 0
eta = 0.1        # learning rate

for step in range(20):
    g = grad(w)
    w = w - eta * g               # w ← w − η∇L
    print(f"step {step:2d}:  w = {w:+.4f}   loss = {loss(w):.4f}")

Run it and $w$ marches steadily toward 0. Now change one line and watch the two failure modes appear on cue. Set eta = 0.001 and $w$ crawls — after 20 steps it has barely moved. Set eta = 1.1 and, because the curvature here is $L'' = 2$ so the ceiling is $2/2 = 1$, you are over the limit: $w$ doesn't shrink, it grows every step and races off to infinity. The threshold from the math is not a suggestion; the code hits it exactly.

How this shows up on the job

You almost never call gradient descent by hand — loss.backward() and optimizer.step() in PyTorch do it for you. What you do spend real time on is the learning rate, because it is the parameter this whole article has been circling.

Tuning the learning rate is the first move, every time. If the loss diverges to NaN in the first few steps, the learning rate is over the curvature ceiling — cut it by 3–10× before touching anything else. If the loss falls but glacially, it is too small. The standard tool for finding the right order of magnitude is the LR range test: sweep the learning rate up over a few hundred steps and plot loss against it; the good value sits just before the point where the loss turns back up.

Warmup and schedules are ravine management. Real training almost never uses a fixed $\eta$. A warmup ramps it from near-zero over the first few hundred steps, because early on the parameters are far from any valley and the curvature is treacherous. After that, a decay schedule (cosine or linear) shrinks it as you approach the bottom, so the big early steps make progress and the small late steps settle in without bouncing.

Gradient clipping is the guardrail against the ceiling. When a single batch produces a freak-large gradient — common in language models — a normal step launches you out of the valley. Clipping caps the gradient's length before the update, which is why clip_grad_norm_ sits in nearly every serious training loop.

"Stochastic" is the word hiding in the acronym. In practice you never compute the gradient over the entire dataset (too expensive); you estimate it from a small mini-batch. That makes each gradient a noisy approximation of the true one — this is stochastic gradient descent (SGD). The noise is not purely a cost: it is exactly the jitter that shakes the optimizer off saddle points and out of sharp, narrow minima that generalize poorly.

Summary

• Learning is descending a loss surface you can only feel locally; the gradient $\nabla L$ is the compass, pointing in the direction of steepest ascent
• The entire method is one line, $w \leftarrow w - \eta \nabla L$: step opposite the gradient, scaled by the learning rate $\eta$
• The learning rate has a hard ceiling set by curvature ($\eta < 2/L''$ on a quadratic) — cross it and the loss diverges; stay far below it and training crawls
• Ravines force plain gradient descent to zigzag; momentum is the standard cure
• In high dimensions the real obstacle is saddle points, not local minima — which is a big part of why training deep networks works despite a non-convex loss

If the calculus underneath the gradient felt shaky, start with Calculus for AI. To see where this one line grows into Adam, momentum, and second-order methods, continue to Beyond SGD.