roformer

Definition

The paper that introduced Rotary Position Embedding. Instead of adding a position vector, RoPE rotates the query and key vectors by an angle proportional to their index, with the effect that their dot product depends only on the relative offset between the two tokens. The scheme is cheap, adds no parameters, and has become the default in most current large language models, including the LLaMA family.