longformer

Definition

A Transformer that replaces dense self-attention with a sliding local window plus a handful of globally attending tokens, making cost scale linearly rather than quadratically in sequence length. Released by the Allen Institute for AI, it let encoder models read whole documents instead of the roughly 512-token windows BERT-style models were stuck with. It belongs with BigBird and Reformer as the early sparse-attention lineage that today's long-context work descends from.