#long-context

9 articles

01 ·Inference & Serving·★ MEMBER·PAPER·6 min read Paper Explained: DeepSeek-V4.1-Flash — How Far Can KV Cache Compression Go? DeepSeek-V4.1-Flash shrinks its global KV cache to 890 bytes per token. We unpack the four ideas behind it (the CED layer split, cross-layer KV sharing in CSA2, FP4 caching, and SWA Bounded Replay) from first principles. 02 ·How Transformers Work·★ MEMBER·PAPER·15 min read Paper Walkthrough: SAS — Training a Sparse-Attention Block Selector End-to-End With the Language Modeling Loss The selector that decides which context blocks a query reads has traditionally been trained by imitating the dense model's attention distribution. SAS injects the selector's scores into the softmax as log-space gates, so the gradient that Top-K used to block flows straight from next-token prediction into the selector. We follow the paper through its four-axis ablation, the gradient formulas behind each choice, and the 10.6–15.5 point GPQA-Diamond gain at a 1024-token budget. 03 ·Inference & Serving·FREE·PAPER·9 min read KV Cache Compression — Quantize, Evict, Share Your Way to Cheaper Long-Context Inference Raise the context length and your concurrency collapses. The culprit is the KV cache, and its size is one plain multiplication. That formula has only three terms you can touch, so compression has only three directions — lower the precision, throw tokens away, share heads. KIVI, H2O, SnapKV, GQA and attention sinks all line up on those three. 04 ·Large Language Models·FREE·PAPER·12 min read Mamba and State Space Models — Handling Sequences Without Attention Attention keeps everything and re-reads it on every step, and the price is quadratic cost in sequence length. State space models take the opposite bet: keep one fixed-size note and update it. Starting from a continuous-time linear system, we trace why S4 can secretly become a convolution, what exactly Mamba made 'selective' in order to give that convolution up, and where the whole approach loses to attention. 05 ·How Transformers Work·★ MEMBER·PAPER·11 min read Paper Explained: Why Gated DeltaNet Survives 4-Bit Quantization — NVFP4 W4A4 in a Hybrid 27B A walkthrough of the paper that tested — and overturned — the belief that the recurrent half of a hybrid LLM is too fragile for 4-bit quantization, and explained mechanistically why log-space gates and the delta rule erase quantization noise. 06 ·Inference & Serving·★ MEMBER·PAPER·12 min read Paper Explained — LatentPress: Feeding Compressed Context Straight to a Frozen LLM, Neither as Text Nor as Pixels A method that writes long conversation histories and documents into continuous memory tokens a frozen LLM reads directly — no text summary, no rendered image. On LongMemEval it beats uncompressed evidence at 7.70x compression. Explained from first principles. 07 ·Inference & Serving·★ MEMBER·PAPER·9 min read Paper Walkthrough: Language Models Can Control Their Own Attention During long-context decoding, a model re-reads its entire KV cache at every step. Declarative Attention has the model announce, inside its own chain-of-thought, where it will look next, and the inference engine builds an attention mask from that text. A walkthrough of the paper that cut attended tokens by 52.0% and 31.1% on off-the-shelf models, zero-shot. 08 ·VLMs & Multimodal·★ MEMBER·PAPER·8 min read Video Understanding from Scratch — From a Pile of Frames to a Sense of Time A video is not just a lot of images. This piece splits video understanding into three questions — which frames to take (sampling), how to mix them (temporal attention), and how to shrink a long video (compression) — and works through each with equations, interactive figures, and code. 09 ·How Transformers Work·★ MEMBER·PAPER·11 min read How Long-Context LLMs Work — From RoPE Interpolation to Ring Attention A "128K context window" is two different walls, knocked down by two unrelated families of tricks. This walks through positional interpolation, NTK-aware scaling and YaRN for the position wall; sliding windows and ring attention for the compute wall; and how to read a needle-in-a-haystack chart without being fooled by it.