Calculus of Variations — What It Means to Differentiate a Function

When the thing you are searching for is a shape rather than a number, differentiation has to be reinvented. Functionals and the first variation, the Euler–Lagrange equation, why the fastest slide is a cycloid, and how all of it lands on the ELBO — the variational problem machine learning runs most.

When You're Searching for a Shape, Not a Number

Ordinary optimization is a hunt for numbers. What learning rate? How many layers? The answer is a number, or a list of numbers, and the recipe is the familiar one: differentiate, set it to zero, solve.

Some problems refuse to fit that mold.

Design a slide. The start and end points are fixed, there's no friction, and you want a ball released at the top to reach the bottom in the least possible time. What shape should the slide be? A straight line? A steep initial drop that builds speed, then a flat run-out?

What you're searching for here isn't a number — it's the curve itself, a function. There are infinitely many candidates, and you can't even line them up and number them. And yet we still want "differentiate and set it to zero." The machinery built to make that possible is the calculus of variations.

Historically it starts in 1696, when Johann Bernoulli posed exactly this slide problem — the brachistochrone — as a public challenge. The leading mathematicians of the day, Newton and Leibniz among them, each solved it their own way. About fifty years later Euler and Lagrange turned those one-off tricks into a general procedure. That procedure is what we now call the calculus of variations.

A Functional Is a Box That Eats a Function and Returns a Number

One piece of vocabulary first: the functional.

An ordinary function takes a number and returns a number — $f(3) = 9$. A functional takes a whole function and returns a number. To keep them apart we write it with square brackets: $J[y]$.

Examples make it click.

• Feed in a curve $y$, get back its length
• Feed in a curve $y$, get back the time a ball takes to slide down it
• Feed in a probability distribution $q$, get back its entropy
• Feed in a predictor $f$, get back its average loss over the data

Look hard at that last one. When we say "minimize the loss" in machine learning, what we're really doing is moving a function $f$ around to minimize a functional. We don't notice because we do it indirectly, by nudging a network's parameters $\theta$ — but the underlying problem is a variational one.

It's worth being precise about where the difficulty comes from. A model with a million parameters still lives in a finite-dimensional search space: a million dimensions, but finite. Open the search up to "any function at all" and the space becomes infinite-dimensional — the value at each point can be chosen independently, so there are as many degrees of freedom as there are points. Enumerating candidates is hopeless, and so is writing down every partial derivative. That gap is why the calculus of variations exists as a subject of its own.

The functionals it handles look like this:

Symbol by symbol: $y(x)$ is the curve we're looking for, $y'(x)$ is its slope, and $L$ — the Lagrangian — is a function that looks at the position, the height and the slope at one point and returns a cost. The integral sums that cost over the whole interval.

Put in words, equation (1) says: every point along the path costs a little something, and the total cost of the path is all of those added up. For the slide, the local cost is the time to cross a sliver of the path; for arc length, it's the length of that sliver.

"Differentiating a Function" Means Wiggling Its Shape

Recall what $f'(x)$ actually measures: nudge the input by $h$, and see how far the output moves. At a minimum, nudging either way produces no first-order change — which is why "derivative equals zero" locates it. Calculus for AI builds that picture up from scratch.

The variational version keeps the idea intact and changes only what gets nudged: a shape instead of a number.

Take a candidate curve $y(x)$ and wiggle it. Write the wiggle pattern as a function $\eta(x)$ (eta), scale it by a knob $\varepsilon$ (epsilon), and form

$$y_\varepsilon(x) = y(x) + \varepsilon\,\eta(x)$$

Put in words: lay a thin bumpy stencil over the original curve. Turning $\varepsilon$ down to zero peels the stencil off and gives the original back.

One condition comes along for the ride: $\eta(a) = \eta(b) = 0$, meaning no wiggling at the endpoints. The top and bottom of the slide are nailed down.

With the stencil fixed, $J[y_\varepsilon]$ becomes a function of the single number $\varepsilon$. The infinite-dimensional problem collapses, one stencil at a time, into ordinary single-variable calculus. The derivative at $\varepsilon = 0$ is the first variation, written $\delta J$:

$$\delta J[y;\eta] = \left.\frac{d}{d\varepsilon} J[y + \varepsilon\eta]\right|_{\varepsilon=0}$$

Which is to say: "if I wiggle this curve with this particular stencil, does the total cost go up or down?" — answered by one number.

So the condition for a minimum reads: $\delta J = 0$ for every possible stencil $\eta$. A shape that no bump can improve is a candidate for the answer.

The word "every" is carrying the weight. Zero response to one stencil isn't enough. Lift the middle of the slide; push the middle down; raise the first half and drop the second; try every bump you can dream up — and the first-order change has to vanish for all of them. Only then is there genuinely nothing left to adjust. Turn it around and it's more useful: if even one wiggle produces a gain, moving the shape that way is guaranteed to improve things. "No improving direction remains" is precisely what "the gradient is zero" means in ordinary parameter training. The stage got bigger; the reasoning did not change at all.