optimization

Definition

The mathematics and practice of adjusting parameters to minimize a loss — in deep learning, almost always stochastic gradient descent and its variants. Learning rate schedules, momentum, weight decay, and gradient clipping are where much of the practical difficulty lives, and a bad optimizer configuration can hide a good architecture. Distinguish this from engineering optimization, meaning making inference faster.

Explainers using this term

• Paper Walkthrough: AutoSaddler — Growing a Harness That Doesn't Break, from Agent Failure LogsAutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces
• TTPO Explained: Training a Model Mid-Exam, With No Answer KeyTTPO: Test-Time Policy Optimization
• Kernel Methods and Gaussian Processes — The Champions Before Neural NetsPractical Bayesian Optimization of Machine Learning Algorithms