Positional Encoding from Scratch — From Absolute Positions to RoPE

A bare Transformer has no idea what word order is. Starting from why position information is needed at all, this article walks through sinusoidal absolute encodings, learned embeddings, and RoPE — the modern LLM standard — showing exactly why rotation encodes relative position.

Attention Is All You Need

The Transformer doesn't know word order

In the previous article (Attention from Scratch) we met self-attention:

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$

Spelled out in words: score how well each word's question ($Q$) matches every other word's label ($K$), convert those scores into shares that add up to one, and use them to blend the contents each word carries ($V$). The $\sqrt{d_k}$ is nothing more than a divisor that keeps the scores from swinging too wildly as the dimension $d_k$ grows.

A surprising property hides in this formula. Shuffle the input words and the output is simply shuffled the same way.

The reason is straightforward. Permute the rows of $X$ and the rows of $Q, K, V$ get permuted identically; the score table $QK^\top$ has its rows and columns permuted together; the final output receives the same permutation. In other words, "the output for word $i$" never consults where that word sat. It only consults which words were present as a set.

So to a bare Transformer, "the dog chases the cat" and "the cat chases the dog" are indistinguishable. An RNN read left to right, so word order was baked into its structure. The Transformer traded that away for the ability to look at every word at once.

The analogy: a round table where everyone speaks simultaneously. You know who said what, but nowhere is there any record of the order in which they spoke. Positional encoding is the job of handing out seat numbers at that table.

Why the naive idea fails

The most obvious approach is to add the position index $m$ straight into the word vector. It breaks for three reasons.

1. The value is unbounded. In a 1,000-word document, position values reach 1,000. Embedding components typically live in roughly the $-1$ to $1$ range; drop a 1,000 in there and the word's meaning is completely washed out.
2. Normalizing doesn't save it. Using $m / L$ (divide by sequence length) keeps you in $0$ to $1$, but now "position 3" means a different number depending on how long the sentence is. Position 3 should be position 3 regardless.
3. One number isn't enough. We represent meaning with hundreds of dimensions. Giving position a single dimension leaves the model no room to use position in nuanced ways.

What we want is a representation that is bounded, unique per position, independent of sequence length, and multi-dimensional.

Absolute positions, take 1: a ruler made of sines and cosines

The original paper by Vaswani et al. went with trigonometric functions. For position $pos$ and dimension index $i$, build this vector and add it to the word embedding:

$$PE_{(pos,\,2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right), \qquad PE_{(pos,\,2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d}}\right)$$

which says: take how far along the word sits ($pos$), divide it by $10000^{2i/d}$, read the result as an angle, and write down its sine and cosine side by side. Here $i$ counts which pair of dimensions you're on, and the bigger that divisor, the less the value budges when the position advances by one.

where $d$ is the embedding dimension. The formula looks imposing, but all it does is line up a set of clock hands spinning at different speeds.

Think of each dimension pair as one hand. The $i=0$ pair is $\sin(pos)$, $\cos(pos)$ — the fastest hand. As $i$ grows, the denominator $10000^{2i/d}$ grows and the hand turns more slowly. As the paper notes, the wavelengths form a geometric progression from $2\pi$ to $10000 \cdot 2\pi$.

It clicks if you see it as a continuous binary counter. In binary, the lowest bit flips every step, the next every two steps, the next every four; read all the bits at once and the number is uniquely determined. The sinusoidal version is the same idea: read the fast hands and the slow hands together and the position is pinned down uniquely. And since every component stays within $-1$ to $1$, failure modes 1 and 2 of the naive idea are solved in one stroke.

The paper also states its reason for choosing this function: "we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset $k$, $PE_{pos+k}$ can be represented as a linear function of $PE_{pos}$."

That sentence is the seed of RoPE. Pull out a single hand (the pair at frequency $\omega$) and you get:

$$\begin{pmatrix} \sin\omega(pos+k) \\ \cos\omega(pos+k) \end{pmatrix} = \begin{pmatrix} \cos\omega k & \sin\omega k \\ -\sin\omega k & \cos\omega k \end{pmatrix} \begin{pmatrix} \sin\omega\,pos \\ \cos\omega\,pos \end{pmatrix}$$

The pair on the left is where that hand points at position $pos+k$, the pair on the right is where it points at position $pos$, and the matrix wedged between them is a rotation by the angle $\omega k$ — which says the amount you turn is fixed by the offset $k$ alone and never depends on where you started.

Moving $k$ steps forward equals rotating by a fixed angle. Rotation has entered the picture.

Absolute positions, take 2: just learn the vectors

There's a simpler option: allocate one vector per position and train it along with everything else. A vector for position 0, one for position 1, and so on — a plain lookup table. BERT and GPT-2 did exactly this.

The original paper compared the two and reported nearly identical results (Table 3, row E). It still chose sinusoidal, citing the possibility of extrapolating to sequences longer than those seen during training.

That points straight at the weakness of the learned approach: positions outside the table do not exist. Feed a 513th token to a model trained on 512 positions and there is no vector to look up. Whatever maximum length you configured becomes a hard ceiling on the model.

The pivot: stop adding, start rotating

Every method so far shares one property: position information is added to the word vector. Meaning and position get blended into a single vector at the input, and we hope attention untangles them later.

RoPE (Rotary Position Embedding, from Su et al.'s RoFormer) takes a different route. Don't add anything. Rotate the Query and Key by an angle proportional to position.

Work in two dimensions. Take a Query vector $q$ at position $m$ and rotate it by $m\theta$:

$$\tilde{q}_m = R(m\theta)\, q, \qquad R(\phi) = \begin{pmatrix} \cos\phi & -\sin\phi \\ \sin\phi & \cos\phi \end{pmatrix}$$

Put plainly, the whole operation is: take the question vector $q$ belonging to the word at position $m$ and spin it by an angle proportional to $m$. $R(\phi)$ is the matrix that turns a vector in the plane by $\phi$ — it changes which way the vector points, never how long it is.

Do the same for a Key at position $n$: $\tilde{k}_n = R(n\theta)\, k$. Now compute what attention actually needs — their inner product:

$$\tilde{q}_m^\top \tilde{k}_n = \left(R(m\theta)q\right)^\top R(n\theta)k = q^\top R(m\theta)^\top R(n\theta) k = q^\top R\big((n-m)\theta\big)\, k$$

The identity in words: how well the Query spun for position $m$ meshes with the Key spun for position $n$ comes out to exactly the same number you would get by taking the unrotated $q$ and $k$ and turning one of them by however many words apart they are, $n-m$.

We used only two properties of rotation matrices: $R(\phi)^\top = R(-\phi)$ and $R(a)R(b) = R(a+b)$.

Look at the result. We built the left side from absolute positions $m$ and $n$, yet only $n-m$ survives on the right. We rotated by absolute position, but the attention score depends purely on relative position. That is the heart of RoPE.

Intuitively: an inner product measures the angle between two vectors (exactly the cosine similarity from Linear Algebra for AI), so if you rotate both by the same rule, the absolute amount of rotation cancels and only the difference remains. It's the same reason the angle between a clock's hour and minute hands doesn't change when you tilt the whole clock.

Check it yourself in the figure below. Rotating just one arrow moves the inner product; rotating both by the same angle leaves it completely unchanged. That invariance is the whole trick behind RoPE.

Real embeddings aren't two-dimensional, so we split the $d$ dimensions into $d/2$ pairs and rotate each pair by its own angle $\theta_i = 10000^{-2i/d}$. Seen this way the lineage is clear: RoPE reuses the same frequency ladder as the sinusoidal method, but rotates instead of adding.

Su et al. further show that under this construction the upper bound of the inner product decays with relative distance — distant tokens tend to influence each other less.

RoPE in code

import numpy as np

def rope(x, base=10000.0):
    """x: (seq_len, d) with d even. Apply to Q and K (never to V)."""
    seq_len, d = x.shape
    pos = np.arange(seq_len)[:, None]        # (seq, 1)
    i = np.arange(d // 2)[None, :]           # (1, d/2)
    theta = base ** (-2.0 * i / d)           # same frequency ladder as sinusoidal
    ang = pos * theta                        # (seq, d/2) rotation angle per pair

    cos, sin = np.cos(ang), np.sin(ang)
    xe, xo = x[:, 0::2], x[:, 1::2]          # the two halves of each pair
    out = np.empty_like(x)
    out[:, 0::2] = xe * cos - xo * sin       # a plain 2-D rotation
    out[:, 1::2] = xe * sin + xo * cos
    return out

# Usage: rotate Q and K right before computing scores
Q, K, V = rope(X @ W_q), rope(X @ W_k), X @ W_v
scores = Q @ K.T / np.sqrt(Q.shape[-1])      # only relative position matters

By the identity derived above, the pair at positions (2, 5) and the pair at (100, 103) produce exactly the same score, up to floating-point error. Same distance, same result, anywhere in the sequence.

Three implementation points matter:

• Rotate Q and K only; leave V alone. Position should influence where you look, not what you carry back.
• Apply it at every layer. The sinusoidal method adds position once at the input; RoPE rotates Q and K at every attention computation, so positional information doesn't fade as it passes through layers.
• Norms are preserved. Rotation is an orthogonal transform, so vector lengths are untouched. Adding position never disturbs the scale — an unglamorous but genuinely useful property.

Other routes to relative position: bias terms

RoPE isn't the only answer. Historically, Shaw et al. came first, adding learned relative-position vectors to the Keys and Values, and T5 took a lighter route: add a learned scalar per distance bucket per head directly to the attention score. ALiBi (Press et al.) goes further still, carrying no positional embedding at all and merely subtracting a distance-proportional penalty $m \cdot |i-j|$ from the score — enough, the paper showed, to train on short sequences and extrapolate to long ones.

Stretching to long contexts

RoPE has its own ceiling. Feed 32K tokens to a model trained on 4K and you hit rotation angles never seen during training, and quality collapses.

The lineage of fixes is worth knowing:

• Position Interpolation (Chen et al.): scale position indices by $L/L'$ to squeeze them back into the trained angular range — treat position 8000 as if it were position 1000. The paper reports long-context capability after a small amount of fine-tuning.
• Changing the base: raise the base 10000 in $\theta_i$ so every hand turns more slowly, which likewise keeps angles inside the familiar range. This originated in the community and was systematized in work such as YaRN.

So when you read that someone "extended a RoPE model's context length," the substance is almost always that they adjusted how finely the rotation angle is stepped.

In practice

1. If you're unsure, use RoPE. As of 2026, the major open LLMs — LLaMA-family, Qwen, Mistral, Gemma — use RoPE. For a new decoder-style model it's the default, and there's rarely a reason to deliberately pick learned absolute positions. You'll still meet learned embeddings when working with encoder-style (BERT-family) models.

2. There are two conventions for pairing dimensions. The original formulation pairs adjacent dimensions (0-1, 2-3, ...), while many implementations pair the first half with the second half (0-(d/2), 1-(d/2+1), ...). They're mathematically equivalent given a consistent permutation of dimensions, but mixing them breaks things silently. If you port weights to another implementation and the outputs are just vaguely wrong, suspect this first.

3. Compute cos/sin in float32. At positions in the tens of thousands the angles get large, and running the trigonometry in bfloat16 loses precision. That's why most implementations keep the position table in float32 even in a mixed-precision model.

4. The KV cache holds already-rotated Keys. RoPE is applied before caching, so a stored K is "rotated for that position." Optimizations that drop part of the cache and repack it effectively shift the positions of the remaining tokens. This is an easy way to break a hand-rolled long-context inference path.

5. How to read "supports 128K context." Whether the model was trained at that length from scratch or stretched there via position interpolation makes a real difference. When stretched, accuracy can degrade in the upper reaches of the advertised window — so measure on your own task rather than trusting the headline number.

Summary

• Self-attention is equivariant to word order, so without externally supplied position information it sees nothing but a bag of words
• Sinusoidal encoding lines up hands of differing frequency to encode position uniquely — and already contained the property "$k$ steps ahead = a fixed rotation"
• Learned embeddings are simple but can never go beyond the positions in their table
• RoPE rotates Q and K in proportion to position; taking the inner product cancels the absolute positions and leaves only the relative one
• In practice, the pairing convention, float32 angle computation, and interaction with the KV cache are where things go wrong

If you want to read the place positional encoding first appeared, head to the paper walkthrough (Reading Attention Is All You Need) — the brief §3.5 is where all of this starts.