decay

Definition

Shrinking a quantity over the course of training. Weight decay is regularization: each update pulls parameters slightly toward zero, discouraging large weights. Learning-rate decay is scheduling: the step size follows a curve, commonly cosine or linear, so updates become finer as training proceeds and the run settles rather than oscillates. Both are among the hyperparameters with the largest effect on final quality.