Hyperparameter Search — Hunches, Grids, and Bayesian Optimization

Gradients tell you nothing about the learning rate, so you have to go looking. Why grid search is weak, why search spaces should be carved on a log scale, what a Bayesian acquisition function is actually counting, and why early stopping beats a cleverer search algorithm — with Optuna code and the traps that bite in production.

Random Search for Hyper-Parameter Optimization

An analogy: an old amp with twelve knobs

Picture a studio at 2 a.m. with an amplifier that has twelve unlabelled knobs. The manual is long gone, nobody knows what any of them do, and checking whether a setting sounds good takes half a day. Trying five positions per knob means five to the twelfth power — over two hundred million combinations. You do not have that many afternoons.

Training a neural network puts you in exactly that room. Gradient descent handles the weights for you. But the settings of gradient descent itself — learning rate, batch size, weight decay, depth, dropout — come with no instructions at all. Those are the hyperparameters, and hunting for a good combination is hyperparameter search.

Where the line falls

The test fits in one sentence: if you can differentiate the loss with respect to it, it is a parameter; if you cannot, it is a hyperparameter.

Differentiating the loss with respect to a weight tells you which way to nudge it, so gradient descent can take over. You cannot differentiate the loss with respect to the learning rate. The only way to learn what a different learning rate would have done is to train again from scratch. That property — one trial costs one full training run — is the whole difficulty of the field.

Hyperparameters fall into four families. Optimization: learning rate, batch size, choice of optimizer, weight decay, warmup length. Model: depth, hidden width, number of heads, dropout rate. Data: augmentation strength, sampling ratios, sequence length. Loss: label smoothing, distillation temperature, auxiliary-loss coefficients.

And one fact matters more than any of the four. The knobs that move the result are a small minority. You may have twelve, but two or three usually account for nearly all the variance, while the rest barely register anywhere inside their sensible range. This lopsidedness — the technical phrase is low effective dimensionality — is the foundation under every method that follows.

Starting with a hunch is not cheating

Turning knobs by hand is still a strong method, for a simple reason: a human sees information an automated search never receives. The search gets one number, the final validation score. You get the shape of the loss curve, and from it you can tell "that is blowing up because the rate is too high" from "that is flat because regularization is too strong."

Two conditions, though. Write everything down, and change one thing at a time. If you move three knobs and the score improves, you have learned nothing about which one did it.

And the first knob is almost always the learning rate. Every other setting fights over a few percentage points; the learning rate decides whether training happens at all.

As the figure shows, the gap between 0.01 and 0.1 matters far more than the gap between 0.01 and 0.02. So before you automate anything, sweep two or three orders of magnitude by hand and find the ceiling where it diverges and the floor where nothing moves. The valley between them is the range you hand to the search. What to do with the rate during training is covered in Learning Rate Schedules.

Why grid search is weak — the real reason

Grid search lays a regular lattice over the space and tries every cell. With $d$ knobs at $k$ settings each that is $k^d$ runs: ten knobs at five settings is about ten million, which rules it out on its own.

But combinatorial blow-up is not the interesting problem. Wasted resolution is.

Suppose you have two knobs, one important and one nearly irrelevant, and a budget of nine runs. A 3×3 grid tries only three distinct values of the knob that matters, because it dutifully spends three settings apiece on the one that does not. Sample nine points at random instead and the important knob gets nine distinct values — three times the resolution for the same money. This is the observation Bergstra and Bengio published in 2012, and combined with the lopsidedness noted above, it explains random search's advantage completely.

You can also estimate random search's hit rate with a one-line calculation.

Here $p$ is how wide you define a "hit" (taking the top 5% of all configurations means $p = 0.05$) and $n$ is the number of trials. The formula just says: take the probability of missing every time, $(1-p)^n$, and subtract it from one. With $p = 0.05$ and $n = 60$ it comes to about 0.95 — so if you only need a configuration in the top 5%, sixty random draws get you there 95% of the time. Clever search earns its keep only past that point, when you are chasing the top 0.1%.

So when is a grid still right? When you are down to two or three knobs whose values are genuinely discrete — three optimizers, four batch sizes — keep the grid. Every cell gets filled, the result reports cleanly as a table, and you can answer "did we try that combination?" on the spot. Once you have four or more continuous knobs, switching to random reaches much further on the same compute. You make this call once, before the search starts, and random is the safe default when you are unsure.

Space design decides most of the outcome

Where you let the search look matters more than which algorithm does the looking.

The big one is the log scale. Draw the learning rate uniformly from 1e-5 to 1e-1 and 90% of your samples land above 1e-2, while almost nothing below 1e-4 is ever tried. The learning rate is a quantity where doubling changes the meaning, so a linear uniform draw treats the decades wildly unequally. Log-uniform gives 1e-5–1e-4, 1e-4–1e-3, and so on equal probability. Weight decay, regularization coefficients, and temperature want the same treatment. Quantities that are genuinely fractions of one — dropout rate, momentum — can stay linear.