minima

Definition

Plural of minimum: points where the loss is lower than everywhere nearby. A deep network's loss surface contains an enormous number of them, and which one training settles into is a substantive question rather than a formality, since they differ in how well they generalize. Much of this work concerns the geometry around the minimum — how wide the basin is — rather than the loss value at the bottom.