linear

Definition

Two senses, both common. Algebraically, a map that preserves addition and scaling — implemented as a matrix multiply, the basic building block of every network. Complexity-wise, cost that grows in proportion to input size: "linear attention" means methods that reduce attention's quadratic dependence on sequence length. Which sense applies is clear from whether the surrounding words are about layers or about cost.

Explainers using this term

• Paper Deep Dive: Your Transformer Can Hold Two Thoughts at Once — Blend Two Texts and the LLM Still Remembers BothYour Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs

In the textbook

• Laid Out in a Grid, It Computes — Linear Algebra