asynchrony

Definition

Letting participants proceed without waiting for each other. In distributed training it removes the synchronisation barrier where every worker stalls for the slowest one, at the cost of stale gradients — updates computed against parameters that have since moved — which changes convergence behaviour and makes runs harder to reproduce. At the kernel level the same idea means overlapping communication with computation so that transfer latency hides behind useful work.