How Long-Context LLMs Work — From RoPE Interpolation to Ring Attention

A "128K context window" is two different walls, knocked down by two unrelated families of tricks. This walks through positional interpolation, NTK-aware scaling and YaRN for the position wall; sliding windows and ring attention for the compute wall; and how to read a needle-in-a-haystack chart without being fooled by it.

Extending Context Window of Large Language Models via Positional Interpolation

What exactly does "128K context" fix?

The model card says "context length: 128K." So what actually happens when you push 128,000 tokens through that model? The answer is that two walls with completely different personalities are being knocked down by two unrelated sets of tricks. Conflate them and every long-context discussion turns to mush.

The first is the position wall. The model was trained at 4K or 8K. Feed it "position 90,000" — a position it has literally never seen — and the scheme for representing position wanders into unseen territory, and the output falls apart. This has nothing to do with compute; it would happen on a machine with infinite memory.

The second is the compute wall. Attention has every token look at every token, so both arithmetic and memory grow with the square of the length. Even if you solved the position wall perfectly, the matrix does not fit on one GPU.

Positional interpolation and YaRN attack the first. Windowing and ring attention attack the second. And needle tests are how you check whether either actually worked. We'll take them in that order.

The metaphor: a 12-inch ruler and a crowded desk

The position wall is a ruler problem. You are a measurer trained only on a 12-inch ruler; reading its marks is muscle memory. One day someone hands you a 36-inch plank. You have two options.

Extrapolate: trust that the marks keep going past 12 at the same spacing, and imagine them onto the plank. It's territory you've never seen, so you'll usually be wrong.

Interpolate: shrink the plank to a third of its size and measure the copy with the ruler you know. Every mark now falls in familiar range — but fine distinctions get crushed. What was a 1mm gap becomes a 0.33mm gap, and two adjacent points start looking like one.

Essentially every long-context positional method is an argument about extrapolating versus interpolating, and about which parts get which treatment. The compute wall, by contrast, is a desk-space problem: you cannot spread everyone's exam papers on a single desk. The two families look nothing alike because the problems don't either.

Recap: RoPE rotates in proportion to position

Nearly every modern open LLM uses RoPE (rotary position embedding). Positional Encoding from Scratch covers it properly; here we need exactly one fact.

RoPE splits the Query and Key vectors into two-dimensional pairs and rotates each pair by an angle proportional to the position $m$. How fast the $i$-th pair turns is set by:

Here $b$ is a constant called the base (10000 in most models), $d$ is the head dimension, $\theta_i$ is the angle added per token, and $\lambda_i$ is how many tokens it takes that pair to complete one full turn — its wavelength. In words, equation (1) says only this: each pair spins at its own rate, and the larger $i$ is, the slower it spins.

The payoff is that when you take the dot product of a Query at position $m$ with a Key at position $n$, $m$ and $n$ individually cancel and only a function of the difference $m-n$ survives. That is why an operation as crude as "renumber the positions" is even meaningful.

Positional interpolation — squeeze the marks inward

The simplest fix is Position Interpolation (PI). Let $L$ be the trained length and $L'$ the target, pick a scale $s = L'/L$, and shrink the position numbers themselves:

In words, equation (2) says: treat position 90,000 as position 2812.5 when $s=32$. The fact that it needn't be an integer is the whole point — RoPE's rotation angle is continuous, so fractional positions work fine. Now every rotation angle lands inside the trained range, and no unseen angle ever appears.

This is the shrink-the-plank move from the metaphor. You have enough marks now, but they mean less. In practice a model right after PI performs worse than before, which is why the original paper proposes it together with a short fine-tuning run.

Read it in wavelengths and the damage becomes obvious

What PI breaks becomes clear the moment you lay out the wavelengths $\lambda_i$ from equation (1). Pairs with small $i$ have wavelengths just a few tokens long, and they are what distinguishes one word from its neighbor. Pairs with large $i$ have wavelengths of tens of thousands of tokens and never complete a single turn within the trained length $L$.

PI shrinks all of them by the same $1/s$. But only the second group was in trouble. Long-wavelength pairs saw only a slice of the angle range during training, so extrapolating pushes them somewhere unseen. Short-wavelength pairs, meanwhile, spun tens of thousands of times during training and have covered the entire angle range — for them there is no unseen territory to extrapolate into. Shrinking them anyway costs you local positional resolution for nothing.

Which hands you the fix. Leave the fast pairs alone; only shrink the slow ones.