partitioning

Definition

Dividing a model, a batch, or optimizer state into disjoint pieces assigned to different devices. The choice of axis determines what must be communicated and when, so partitioning strategy — not raw FLOPs — is usually what decides whether a distributed run scales. Parallelism names the outcome of running things at once; partitioning names the concrete assignment that makes it possible, and the two words are often used loosely as if interchangeable.