saddle

Definition

A saddle point is a stationary point where the gradient vanishes but the point is neither a minimum nor a maximum: loss decreases along some directions and increases along others. In high-dimensional non-convex optimization saddle points vastly outnumber local minima, and an influential line of work argues they, not bad minima, are what stalls training. The practical trouble is the flat region around them, where gradients are tiny and progress crawls.

Explainers using this term

• Convexity and Optimization — Why Deep Learning Works Even Though It Isn't ConvexIdentifying and attacking the saddle point problem in high-dimensional non-convex optimization