sinks

Definition

Tokens that soak up a large share of attention weight regardless of their content — the phenomenon named attention sinks. The first few tokens of a sequence typically play this role, apparently serving as a dumping ground for probability mass the softmax must assign somewhere. The practical consequence appears in streaming inference: evict those initial tokens from the KV cache and quality collapses, so implementations keep them pinned.