Students Professional
JA EN

#ring-attention

1 articles

01 ·How Transformers Work·★ MEMBER·PAPER·11 min read How Long-Context LLMs Work — From RoPE Interpolation to Ring Attention A "128K context window" is two different walls, knocked down by two unrelated families of tricks. This walks through positional interpolation, NTK-aware scaling and YaRN for the position wall; sliding windows and ring attention for the compute wall; and how to read a needle-in-a-haystack chart without being fooled by it.