restarts

Definition

Resetting the learning rate — and sometimes the optimiser's momentum state — back toward its initial value partway through training, so the search resumes with energy. Done periodically, as in SGDR, it lets the optimiser escape a basin and pass through several good solutions rather than settling in the first one. Nothing is thrown away: the weights carry over, only the step size is renewed.