large-batch

Definition

Training in a regime where each gradient step aggregates a very large number of samples or tokens, so that many accelerators stay saturated. Large batches give cleaner gradients and scale well across devices, but they require rescaling the learning rate and warming up, and past a critical size extra examples buy little additional progress per step. The phrase signals that a result depends on this regime, not on a single-GPU setup.