Jacobians and Hessians — Multivariable Calculus, Drawn

The Jacobian is a magnifying glass at a point; the Hessian is how sharply the ground curves there. From local linearization to what eigenvalues say about the terrain, why Newton's method is fast on paper but absent in practice, and how to get Hessian information in a billion dimensions without ever building the matrix.

The Earth Is Round, but the Street Looks Flat

The Earth is a sphere. The road outside your house still looks flat, and you can walk it believing that without ever running into trouble. Anything curved can be swapped for something straight if you crop the view tightly enough. That, boiled down, is all differentiation does.

In one variable, the approximation around a point $x$ looks like this:

$$f(x+h) \approx f(x) + f'(x)\,h$$

Put in words: nudge the input by $h$ and the output moves by that same $h$, scaled by $f'(x)$. A derivative is a local magnification factor. Calculus for AI builds that view up carefully; this article picks up where it leaves off.

The catch is that real functions don't take one number in and hand one number back. The input is hundreds of millions of parameters; the output is a logit vector tens of thousands wide. One magnification factor isn't enough. You need one for every pairing of an input with an output. That table is the Jacobian. Differentiate once more and you get a second table — how the magnification itself changes as you move. That one is the Hessian.

When Inputs and Outputs Are Both Plural, the Derivative Becomes a Table

Take a function $f:\mathbb{R}^n \to \mathbb{R}^m$ that eats $n$ numbers and returns $m$. The Jacobian is the $m \times n$ matrix holding every partial derivative:

Equation (1) is a table, which says, in words: if I turn input knob $j$ a little, how much does output $i$ move? — asked once for every pairing. The $\partial$ symbol means "the rate of change when this one knob turns and all the others stay pinned."

The trick to reading it: rows are outputs, columns are inputs. Read row $i$ across and you have the gradient of the single output $f_i$. When there's only one output, $J$ is a single row — the gradient itself. The gradient is just a Jacobian with one row.

It helps to have a feel for the size too. A single linear layer mapping 4096 dimensions to 4096 has a Jacobian with 16 million entries. Stack ten layers and that's ten such tables. So the working rule is that you never build the table: what you actually need is never the matrix, only the matrix applied to a vector. That distinction pays off later.

One concrete example. For the polar-to-Cartesian map $x = r\cos\theta,\ y = r\sin\theta$, the Jacobian is

$$J = \begin{pmatrix} \cos\theta & -r\sin\theta \\ \sin\theta & r\cos\theta \end{pmatrix}$$

The first column says which way the point travels when you stretch $r$ (radially outward). The second says which way it travels when you turn $\theta$ (along the circle's tangent, and faster the larger $r$ is). Each column is literally the direction of motion produced by turning that one knob.

The Jacobian Is a Magnifying Glass at a Point

What a Jacobian really is: a linear stand-in that's only valid in the neighborhood of $x$.

Equation (2), which says in words: near $x$, this twisting, curving function is indistinguishable from "multiply by the matrix $J$." Here $\delta$ is a small displacement vector and $J\delta$ is an ordinary matrix–vector product. Local linearization is the name for exactly this.

Multiplying by a matrix is, geometrically, "stretch, squash, rotate." So a small circle drawn around $x$ comes out the other side as an ellipse. Which directions stretch and which collapse is entirely decided by the Jacobian.

How near "near" has to be depends on the function. Push $\delta$ far enough and the approximation breaks; the size of that error is governed by the second derivative, which is where we're headed. As an aside, a network built purely from ReLUs is piecewise linear, so as long as you stay inside the same linear region, Equation (2) is not an approximation at all — it is exact. Cross a kink and $J$ abruptly becomes a different matrix.

The volume scaling factor also falls out of the same table, as a single number: the determinant $|\det J|$. The absolute value is there because the sign carries separate information — a negative determinant means the map flips orientation, turning the neighborhood into its mirror image on the way through. If $\det J$ is zero, a dimension has collapsed at that point and information is being destroyed. In the polar example $\det J = r$, which collapses only at the origin — the same reason maps misbehave around the poles. This volume factor carries straight over into changes of variable for probability distributions, which is why normalizing flows, a family of generative models, compute $\log|\det J|$ at every step.