ring

Definition

A topology where devices are arranged in a loop and each passes data to its neighbour. Ring Attention gives every GPU one block of the sequence, then rotates key and value blocks around the ring while attention is computed locally, so a sequence far too long for any single device's memory can still be processed. Because the passing overlaps with computation, adding devices adds context length nearly for free.