Students Professional
JA EN

#rope

2 articles

01 ·How Transformers Work·★ MEMBER·PAPER·11 min read How Long-Context LLMs Work — From RoPE Interpolation to Ring Attention A "128K context window" is two different walls, knocked down by two unrelated families of tricks. This walks through positional interpolation, NTK-aware scaling and YaRN for the position wall; sliding windows and ring attention for the compute wall; and how to read a needle-in-a-haystack chart without being fooled by it. 02 ·How Transformers Work·FREE·PAPER·10 min read Positional Encoding from Scratch — From Absolute Positions to RoPE A bare Transformer has no idea what word order is. Starting from why position information is needed at all, this article walks through sinusoidal absolute encodings, learned embeddings, and RoPE — the modern LLM standard — showing exactly why rotation encodes relative position.