batch

Definition

The group of examples processed together in one forward and backward pass. Batch size is a first-order hyperparameter: larger batches give lower-variance gradient estimates and better hardware utilisation but cost memory and usually require rescaling the learning rate. It sits between full-batch gradient descent, which uses the whole dataset per step, and fully online updates on single examples.

Explainers using this term

• A History of Normalization Layers — From BatchNorm to RMSNormBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift