Calculus for AI — The Gradient Is an Arrow Saying Which Way Is Better

No epsilon-delta limits, no integration by parts. Training is measuring a slope and stepping the other way. A derivative is a multiplier, a gradient is a list of slopes, the chain rule is multiplication — and Jacobians and Hessians only need to be recognised, not computed.

The goal: being able to read this equation

Every paper about training contains this line in some disguise.

It is one line of bookkeeping, which says: from where the parameters are now, take a small step in the direction opposite to whichever way the loss increases fastest. $\theta$ ("theta") is every parameter in the model, $t$ counts steps, $\eta$ ("eta") sets the step size, $\mathcal{L}$ is the loss, and $\nabla$ ("nabla") is the subject of this article. Swap every symbol for an ordinary word and the line reads: new settings = current settings − (how big a step) × (the direction that makes things worse).

The amount of calculus required is startlingly small. Formal limits and integration techniques never show up. You need one intuition — a derivative is a slope — plus the vocabulary for extending it to many variables at once.

A derivative is a multiplier, and nothing more

$$\frac{df}{dx}(x) = \lim_{h \to 0}\frac{f(x+h) - f(x)}{h}$$

Here $h$ is a tiny number and $\lim_{h\to 0}$ means "where this ratio heads as $h$ shrinks toward zero".

Put in words: nudge the input by $h$ and ask by what factor the output moves. The numerator is the change in output, the denominator the change in input, and the derivative is their ratio. A slope of 0.5 means nudging the input by 0.01 moves the output by 0.005. A derivative is that local multiplier — there is no deeper content.

The sign carries the weight. Positive slope: increasing the input increases the output. Negative: increasing it decreases the output. To go down, move against the slope. That is the entirety of training.

Partial derivatives: hold everything else still, turn one knob

One knob is easy; real models have millions to billions. So you freeze every other knob and wiggle exactly one, measuring the slope along that direction. That is a partial derivative, written with $\partial$ ("round d") instead of $d$.

$$\frac{\partial L}{\partial \theta_i}$$

The symbol is really a question, which says: how much does the loss change if I nudge only knob number $i$? The $\partial$ on top is the change in the loss, the $\partial$ underneath is the change in that one knob, and the subscript $i$ names which knob. It is the mixing-desk move — push one fader, listen for the difference. Nothing new has been introduced; it is a slope with a promise attached that everything else stays put.

The gradient: every slope, lined up

Compute one partial derivative per knob and stack them in order.

Said in words: the effect of turning each knob individually, simply listed in order. The brackets are the list, each entry inside is one knob's slope, and $n$ is however many knobs the model has. It is not a new concept, it is an inventory of partial derivatives. Seven billion parameters means a gradient with seven billion numbers in it.

But lining them up creates properties the plain inventory did not have:

• The gradient points in the direction of steepest increase of the loss
• The length of the gradient is how steep that increase is

To descend, you go the other way — which is where the minus sign in equation (1) comes from. In Linear Algebra for AI a vector was a coordinate holding meaning; a gradient is an arrow living in that same space.

The chain rule: multipliers compose by multiplying

If deep learning has one genuine trick, this is it.

The setup is a three-stage relationship: changing $x$ changes $y$, and changing $y$ changes $z$.

What it says is: multipliers compose by multiplication. Move $x$ by 1 and $y$ moves by 3; move $y$ by 1 and $z$ moves by 2; therefore moving $x$ by 1 moves $z$ by 6. Identical to gear ratios in a meshed train of gears.

A neural network is a stack of layers, which is to say a composed function — dozens of meshed gears. So the influence of an early parameter on the final output is just the product of every multiplier along the way. Executing that product from the output end backwards is backpropagation, laid out in Backpropagation From Scratch.

Being a product has consequences you can predict from the arithmetic alone. If the factors along the chain are mostly below 1, the product collapses exponentially toward zero (vanishing gradients); above 1, it explodes. Read ReLU, residual connections and normalisation layers as tools for protecting that chain of multiplications, and a lot of architectural choices line up behind a single motive.

Jacobians and Hessians: recognise them, don't compute them

Both names appear constantly in papers. You never have to evaluate one; you only have to know what table it is.

The Jacobian appears when both input and output are vectors — it is the exhaustive table of partial derivatives.

$$J_{ij} = \frac{\partial f_i}{\partial x_j}$$

In plain terms: how much output $i$ moves when input $j$ is nudged, tabulated for every pair. A gradient is the special case where the output is a single number. Backpropagation is exactly the process of multiplying such Jacobians by a vector, working from the output end inward — and implementations compute the product without ever forming the table.

The Hessian collects second derivatives, the slope of the slope.

$$H_{ij} = \frac{\partial^2 L}{\partial \theta_i \partial \theta_j}$$

The $\partial^2$ is a slope of a slope, and the indices $i$ and $j$ pick out a pair of knobs. Read one cell at a time, which says: how the steepness along knob $i$ changes when knob $j$ is turned. Taken as a whole the table is curvature: how sharply the valley bends. If the first derivative tells you which way is down, the second tells you how fast that downhill is turning. Knowing curvature would in principle let an optimizer pick its own step size.

So why is it not used? With $n$ parameters the Hessian is $n \times n$. At a billion parameters the table alone is astronomically large. The per-parameter step sizes in optimizers like Adam are not the Hessian; they come from a running average of squared gradients, a cheap stand-in for curvature.

Convexity: why everyone wishes the loss were convex

$$f(\lambda x + (1-\lambda)y) \le \lambda f(x) + (1-\lambda)f(y)$$

$\lambda$ is a number between 0 and 1; the left side is the function's value at a point between $x$ and $y$, the right side is the height of the straight line joining them.

What this says is: any chord drawn between two points on the graph stays above the graph. A bowl. One valley, no others.

Why does that help? Every local minimum is the global minimum. Reach a point where the slope is zero and you can declare it the answer. Initial values and random seeds do not change the outcome. Linear and logistic regression live here, which is exactly why they reproduce the same solution every run.

Deep learning losses are not convex. There are many valleys, along with saddle points (downhill in one direction, uphill in another) and broad plateaus where the slope nearly vanishes. That is why initialisation and data ordering change your results. It works well enough in practice anyway, and the honest criterion in production is never "did we reach the global optimum" but "is validation performance good enough" (Overfitting and Evaluation Design).

Three things that matter in practice

1. Non-differentiable points do not stop training ReLU has a corner at zero where no slope is defined. Implementations return 0 or 1 there by convention. Landing exactly on zero is vanishingly unlikely and causes no trouble in practice, so "technically non-differentiable" is not a reason to hesitate.

2. Watch the gradient norm The length of the gradient is a health check for training. Spiking means explosion; sitting near zero means vanishing gradients or a plateau. Looking at the loss alone will not distinguish those two.

3. Autodiff is automated chain rule, not magic PyTorch and JAX apply the chain rule mechanically to the graph recorded during the forward pass. Read detach() and no_grad() as declarations of "cut the chain here" and you can trace why gradients stopped flowing.

Summary

• A derivative is a local multiplier; read its sign and step the other way — that is training
• A partial derivative holds everything else still; the gradient is one per knob, listed in order
• The gradient points uphill and its length is the steepness, which is why the update rule carries a minus sign
• The chain rule multiplies multipliers, so gradients vanish or explode; ReLU and residual connections exist to protect that chain
• The Jacobian tabulates all partial derivatives, the Hessian tabulates curvature; both are too large to build, so we approximate
• Convex means local equals global; deep learning is non-convex, so validation performance is the only verdict that counts

Next we re-read the model's output in another language entirely — probability (Probability and Statistics for AI) — where the question of why loss functions have the shapes they do finally gets answered.