positional

Definition

Relating to token order. Self-attention is permutation-invariant on its own, so order has to be injected explicitly through positional encoding. Designs range from added absolute embeddings to relative schemes such as RoPE, which encodes position by rotating query and key vectors. This choice is not a detail: how positions are represented determines how gracefully a model generalizes to sequences longer than it saw in training.

Explainers using this term

• How Long-Context LLMs Work — From RoPE Interpolation to Ring AttentionExtending Context Window of Large Language Models via Positional Interpolation