linear

Definition

Two senses, both common. Algebraically, a map that preserves addition and scaling — implemented as a matrix multiply, the basic building block of every network. Complexity-wise, cost that grows in proportion to input size: "linear attention" means methods that reduce attention's quadratic dependence on sequence length. Which sense applies is clear from whether the surrounding words are about layers or about cost.