parallelism

Definition

Splitting computation across devices so that models too large or slow for one accelerator become tractable. The axes differ in what gets divided: data parallelism replicates the model and splits the batch, tensor parallelism splits individual matrix operations, pipeline parallelism assigns different layers to different devices, and expert or sequence parallelism split along other dimensions. Real systems combine several, because each axis has its own communication volume and its own idle-time failure mode.