optimizers

Definition

The rule that turns gradients into weight updates — SGD, SGD with momentum, Adam, AdamW and relatives — differing in convergence speed, sensitivity to learning rate, and memory cost. Adam-style methods keep first- and second-moment estimates per parameter, so optimizer state can rival the model itself in size. That overhead is why compressing or sharding it is its own research line at scale.