Loss Functions and Optimization — How a Model Learns From Being Wrong

Why MSE and cross-entropy have the shapes they do, what the gradient actually points at, and one step of gradient descent taken apart with equations, a draggable figure, and ten lines of numpy — divergence included.

The metaphor: downhill in thick fog, feeling for the slope

You are on a mountainside in fog so thick you can see one metre ahead. Your goal is the valley floor. You have no map. The only information available is the slope under your feet.

There is exactly one thing you can do: feel around, find the direction that drops most steeply, take a step. Feel again, step again. Primitive — but repeat it and you close in on the bottom.

Optimization in machine learning is that walk. Altitude is the loss: how wrong the model currently is. The slope under your feet is the gradient. The size of each step is the learning rate. What the previous article (What Machine Learning Really Is) called "turning the knobs" becomes a concrete procedure here.

The loss function: collapsing wrongness into one number

Before anything can be learned, "how wrong are we?" has to become a single number. With more than one number you cannot even say whether a change was an improvement. The function that produces it is the loss function.

The default for regression: mean squared error

When the target is a continuous quantity — a rent, a temperature, a delivery time — this is the standard choice.

$$L_{\mathrm{MSE}} = \frac{1}{N}\sum_{i=1}^{N}\left(y_i - \hat{y}_i\right)^2$$

Symbol by symbol: $N$ is the number of data points, $y_i$ is the true value of the $i$-th one, $\hat{y}_i$ ("y-hat") is the prediction for that same point, and $\sum$ means "add this up over all the data". The whole equation says: square the gap between prediction and truth, then average over every example.

Put in words, it answers a single question: on a typical example, how far off is this model — with the big misses counted extra? Lower is better, and nothing else about the data survives that compression.

Why square it? Squaring erases the sign, so being 3 too high and 3 too low count the same. It punishes large misses disproportionately — double the error, quadruple the loss. And, most usefully in practice, a square differentiates cleanly, which makes slopes easy to compute. Mean absolute error (MAE), which uses absolute values, is far less sensitive to outliers but also less eager to crush big failures. Rough guide: if outliers are measurement noise, MAE; if outliers are the events you care about, MSE.

The default for classification: cross-entropy

When the answer is a category — cat or dog, spam or not — the model outputs a probability per class. Cross-entropy scores those probabilities.

$$L_{\mathrm{CE}} = -\frac{1}{N}\sum_{i=1}^{N}\log p_i$$

Here $p_i$ is the probability the model assigned to the correct class for the $i$-th example, a number between 0 and 1. $\log$ is the natural logarithm, and the leading minus keeps the loss positive.

Strip out the notation and you are left with a rule which says: the less probability you handed to the right answer, the more you pay — and the price climbs ever faster as that probability approaches zero.

Plug in numbers and the intent is obvious. Assign 0.9 to the correct class and you pay $-\log 0.9 \approx 0.11$, almost nothing. Assign 0.5 and you pay about 0.69. Assign 0.01 and the penalty jumps to about 4.6, running off to infinity as the probability approaches zero. Cross-entropy hates being confidently wrong.

Why a logarithm? Statistically, the original goal is to maximize the likelihood — the probability of having observed your data. Likelihoods multiply, and products are awkward, so you take logs to turn them into sums, then flip the sign to turn maximization into minimization. Out falls the formula above. Minimizing cross-entropy is maximum likelihood estimation, and that equivalence pays off later.

The gradient: which way is down?

Nudge one parameter slightly upward. If the loss rose, turn it the other way; if it fell, keep going. That ratio — how much the loss changes per unit of nudge — is the derivative. Collect the derivative of every parameter into a vector and you have the gradient, written $\nabla L$.

There is one property to internalize: the gradient points in the direction of steepest increase. So to go down, you move in the direction opposite to it. That is where the minus sign below comes from.

Gradient descent: one step of learning

$\theta$ ("theta") is the knob being turned. $\eta$ ("eta") is the learning rate, a small positive number such as 0.1 or 0.001 that sets the step size. $\frac{\partial L}{\partial \theta}$ is the partial derivative of the loss with respect to that knob — the slope along its direction. The minus sign means "walk against the slope".

Said in words: check which way this knob makes the loss go up, then turn it a little the other way. "Which way" is the derivative, "a little" is $\eta$, and "the other way" is the minus sign. Nothing more is hidden in the line.

Learning is this one line, run tens of thousands or billions of times. Deep learning does not change the structure at all: the only differences are that the knobs number in the billions and that computing the slopes needs a trick called backpropagation.

What happens when the learning rate is too large

Setting $\eta$ is the first wall almost everyone hits.

Too small, and the direction is right but the steps are so tiny you never arrive. The loss curve looks nearly flat, and it is easy to misdiagnose this as "the model can't learn".

Too large is far more dramatic. You overshoot the bottom and land on the opposite slope, where the gradient is steeper still, so the next jump is bigger. The swing grows each round, the loss climbs, and eventually the numbers overflow into NaN (not a number). That is divergence.

Easier to feel than to read about. Push the slider up and watch the ball sail past the valley, oscillate, then fly apart.

When your loss turns into NaN, cut the learning rate by ten and rerun. That reflex will save you hours.

Valleys are not one-dimensional

Everything so far assumed a single knob. Real models have many, and the loss becomes a surface — you are descending a contour map, not a line.

In a long, narrow valley the gradient points at the walls rather than along the valley, so the path zig-zags. Momentum damps this by carrying some of the previous direction into the next step: give the ball mass and the side-to-side components cancel while motion along the valley accumulates. Modern optimizers such as Adam add per-parameter learning rate scaling on top of that.

You will often hear that local minima are the great enemy. In spaces with millions of parameters, though, points that are a minimum in every direction are far rarer than saddle points — down in some directions, up in others — and than broad plateaus where the slope nearly vanishes. When training stalls in practice, plateaus and the learning rate are better suspects.

Gradient descent in ten lines

A real implementation: linear regression ($\hat{y} = Xw$) trained with MSE.

import numpy as np

def train(X, y, lr=0.1, steps=200):
    w = np.zeros(X.shape[1])                  # start every knob at zero
    for _ in range(steps):
        pred = X @ w                          # current predictions
        grad = 2 * X.T @ (pred - y) / len(y)  # gradient of MSE, straight from the formula
        w -= lr * grad                        # one step against the slope
    return w

The grad line is the derivative of MSE; w -= lr * grad is the update equation verbatim. What a framework's autodiff buys you is the grad line — the skeleton never leaves these five lines. Set lr=10 and you can reproduce divergence on your own machine in about thirty steps.

SGD: looking at everything before moving is too slow

The code above computes the gradient over the entire dataset before taking a single step (batch gradient descent). With a million rows, one step costs a million evaluations. That does not scale.

So instead you take one example — or a mini-batch of 32 to 1024 — compute the gradient from just that, and step immediately. This is stochastic gradient descent (SGD).

The intuition: stop drawing an accurate map before each step and instead feel around roughly and move, hundreds of times over. Each direction is noisier, but for the same compute you take orders of magnitude more steps, so you reach the valley floor sooner. The noise even helps at times, jostling parameters out of shallow dips. Essentially every modern training loop is mini-batch based.

Four things that matter in practice

1. Read the loss curve first Judge progress by the loss before accuracy. Spikes and blow-ups mean the learning rate; total flatness usually means input scaling or how the labels were built.

2. Search the learning rate by factors of ten 0.1 → 0.01 → 0.001. Trying one value and concluding "this model doesn't learn" is the most common self-inflicted wound in the field.

3. Standardize your inputs If feature A ranges over 0–1 and feature B over 0–100000, the loss surface becomes a knife-edge ravine and convergence crawls. Rescaling to zero mean and unit variance rounds out the valley, and often the same learning rate suddenly works.

4. Never hand-roll cross-entropy $\log 0$ runs off to infinity, so computing probabilities and then taking their log is numerically fragile. Use the framework's fused, stable version that accepts raw logits.

Summary

• A loss function compresses "how wrong we are" into one number: MSE for regression, cross-entropy for classification
• Cross-entropy punishes confident mistakes brutally — and is exactly maximum likelihood estimation
• The gradient points uphill, so you step in the opposite direction, scaled by the learning rate
• Too large a learning rate diverges, too small never arrives; on NaN, cut the learning rate first
• Real training uses mini-batch SGD: noisier steps, vastly more of them, faster progress

Next: what happens when this optimization works too well and the model fits the training data perfectly — overfitting, and the evaluation design that exposes it (Overfitting and Evaluation Design).