#long-context
6 articles
01
·Large Language Models·FREE·PAPER·12 min read
Mamba and State Space Models — Handling Sequences Without Attention
Attention keeps everything and re-reads it on every step, and the price is quadratic cost in sequence length. State space models take the opposite bet: keep one fixed-size note and update it. Starting from a continuous-time linear system, we trace why S4 can secretly become a convolution, what exactly Mamba made 'selective' in order to give that convolution up, and where the whole approach loses to attention.
02
·How Transformers Work·★ MEMBER·PAPER·10 min read
Paper Explained: Why Gated DeltaNet Survives 4-Bit Quantization — NVFP4 W4A4 in a Hybrid 27B
A walkthrough of the paper that tested — and overturned — the belief that the recurrent half of a hybrid LLM is too fragile for 4-bit quantization, and explained mechanistically why log-space gates and the delta rule erase quantization noise.
03
·Inference & Serving·★ MEMBER·PAPER·12 min read
Paper Explained — LatentPress: Feeding Compressed Context Straight to a Frozen LLM, Neither as Text Nor as Pixels
A method that writes long conversation histories and documents into continuous memory tokens a frozen LLM reads directly — no text summary, no rendered image. On LongMemEval it beats uncompressed evidence at 7.70x compression. Explained from first principles.
04
·Inference & Serving·★ MEMBER·PAPER·9 min read
Paper Walkthrough: Language Models Can Control Their Own Attention
During long-context decoding, a model re-reads its entire KV cache at every step. Declarative Attention has the model announce, inside its own chain-of-thought, where it will look next, and the inference engine builds an attention mask from that text. A walkthrough of the paper that cut attended tokens by 52.0% and 31.1% on off-the-shelf models, zero-shot.
05
·VLMs & Multimodal·★ MEMBER·PAPER·8 min read
Video Understanding from Scratch — From a Pile of Frames to a Sense of Time
A video is not just a lot of images. This piece splits video understanding into three questions — which frames to take (sampling), how to mix them (temporal attention), and how to shrink a long video (compression) — and works through each with equations, interactive figures, and code.
06
·How Transformers Work·★ MEMBER·PAPER·11 min read
How Long-Context LLMs Work — From RoPE Interpolation to Ring Attention
A "128K context window" is two different walls, knocked down by two unrelated families of tricks. This walks through positional interpolation, NTK-aware scaling and YaRN for the position wall; sliding windows and ring attention for the compute wall; and how to read a needle-in-a-haystack chart without being fooled by it.