rotary

Definition

Refers to encoding information by rotating vectors. In rotary position embedding, each query and key is rotated by an angle proportional to its index, so the attention dot product becomes a function of the offset between positions alone. Unlike additive position embeddings, rotation leaves vector norms untouched and introduces no learned parameters, part of why it extends more gracefully to longer contexts.