#sparse-attention

4 articles

01 ·Inference & Serving·★ MEMBER·PAPER·6 min read Paper Explained: DeepSeek-V4.1-Flash — How Far Can KV Cache Compression Go? DeepSeek-V4.1-Flash shrinks its global KV cache to 890 bytes per token. We unpack the four ideas behind it (the CED layer split, cross-layer KV sharing in CSA2, FP4 caching, and SWA Bounded Replay) from first principles. 02 ·How Transformers Work·★ MEMBER·PAPER·15 min read Paper Walkthrough: SAS — Training a Sparse-Attention Block Selector End-to-End With the Language Modeling Loss The selector that decides which context blocks a query reads has traditionally been trained by imitating the dense model's attention distribution. SAS injects the selector's scores into the softmax as log-space gates, so the gradient that Top-K used to block flows straight from next-token prediction into the selector. We follow the paper through its four-axis ablation, the gradient formulas behind each choice, and the 10.6–15.5 point GPQA-Diamond gain at a 1024-token budget. 03 ·Inference & Serving·★ MEMBER·PAPER·9 min read Paper Walkthrough: Language Models Can Control Their Own Attention During long-context decoding, a model re-reads its entire KV cache at every step. Declarative Attention has the model announce, inside its own chain-of-thought, where it will look next, and the inference engine builds an attention mask from that text. A walkthrough of the paper that cut attended tokens by 52.0% and 31.1% on off-the-shelf models, zero-shot. 04 ·How Transformers Work·★ MEMBER·PAPER·16 min read Qwen3.8-Next design report explained (arXiv:2608.30320) A ground-up read of the Qwen3.8-Flash-Next report: 125B total parameters with 6B activated, the GDN hybrid, Qwen Sparse Attention (7.6x faster prefill at 1M context), the Gated Residual and n-gram embeddings — judged on loss, cost and training stability.