#linear-attention

3 articles

01 ·Time Series·★ MEMBER·PAPER·7 min read Paper Explained: STEPQuant — When and Where Quantization Errors Matter in Delta-Rule Recurrent States Linear-attention states have a fixed size, yet under concurrent serving they can outgrow the model weights. Naive low-bit quantization lets errors pile up with every update. STEPQuant spends bits and scales according to how long an error survives and which row it lands on. Explained from scratch. 02 ·How Transformers Work·★ MEMBER·PAPER·11 min read Paper Explained: Why Gated DeltaNet Survives 4-Bit Quantization — NVFP4 W4A4 in a Hybrid 27B A walkthrough of the paper that tested — and overturned — the belief that the recurrent half of a hybrid LLM is too fragile for 4-bit quantization, and explained mechanistically why log-space gates and the delta rule erase quantization noise. 03 ·How Transformers Work·★ MEMBER·PAPER·16 min read Qwen3.8-Next design report explained (arXiv:2608.30320) A ground-up read of the Qwen3.8-Flash-Next report: 125B total parameters with 6B activated, the GDN hybrid, Qwen Sparse Attention (7.6x faster prefill at 1M context), the Gated Residual and n-gram embeddings — judged on loss, cost and training stability.