#sparse-attention
2 articles
01
·Inference & Serving·★ MEMBER·PAPER·9 min read
Paper Walkthrough: Language Models Can Control Their Own Attention
During long-context decoding, a model re-reads its entire KV cache at every step. Declarative Attention has the model announce, inside its own chain-of-thought, where it will look next, and the inference engine builds an attention mask from that text. A walkthrough of the paper that cut attended tokens by 52.0% and 31.1% on off-the-shelf models, zero-shot.
02
·How Transformers Work·★ MEMBER·PAPER·16 min read
Paper Walkthrough: Designing Qwen3.8-Next — Accuracy, Efficiency and Stability as One Problem
A ground-up read of the Qwen3.8-Flash-Next design report: the GDN hybrid, Qwen Sparse Attention, the Gated Residual and n-gram embeddings, judged the way the paper judges them — on loss, on cost, and on stability.