sgdr

Definition

Stochastic gradient descent with warm restarts: the learning rate decays along a cosine curve, then jumps back up at the end of each cycle instead of continuing to zero. The restart kicks the optimiser out of one basin so it can explore another, and the snapshot at each cycle's end can be ensembled for free. Distinct from warmup, which ramps the rate up once at the very start of training.