Paper Explained: STEPQuant — When and Where Quantization Errors Matter in Delta-Rule Recurrent States

Linear-attention states have a fixed size, yet under concurrent serving they can outgrow the model weights. Naive low-bit quantization lets errors pile up with every update. STEPQuant spends bits and scales according to how long an error survives and which row it lands on. Explained from scratch.

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

Fixed size, so why does memory run out?

The paper covered here is "STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization" (Bingchen Yao et al. / arXiv:2609.38169, 2026-09-29).

Here is the abstract in plain terms. Linear attention swaps the ever-growing KV cache for a fixed-size recurrent state, but under concurrent serving these states can become a major memory bottleneck. Quantizing them straight to low precision badly hurts accuracy, because the errors propagate through each state update. The authors find that the damage depends on two dimensions. Temporally, errors in long-lived memory persist across many decoding steps. Spatially, errors in different key rows affect the output differently, and state magnitudes vary widely along both rows and columns. Their method, STEPQuant, is a post-training quantization framework that allocates precision by error magnitude and memory lifetime, and jointly fits key-row and value-column scales. On Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct it closely matches FP32-state accuracy at a nominal 6 bits and beats uniform INT8 in its 4-bit configuration. Integrated into SGLang, the 6-bit version compresses the recurrent state by more than 5x and cuts total serving memory by up to 68.7%.

"If the state is fixed-size, how can it grow?" Because every request needs its own state. In SGLang's official setup, Qwen's FP32 state pool outgrows its BF16 weights at 70 concurrent requests (§1, Fig. 1a). If the KV cache is new to you, Understanding the KV Cache is a good warm-up.

An analogy: a notebook recopied every night

Picture the recurrent state as a notebook. Quantization means recopying that notebook every night onto paper with a coarse grid. The next morning you write on top of the slightly-off copy, and that night you recopy it again, so errors stack on errors. And not all errors are equal: a smudge on a scratch note is gone tomorrow, while a smudge on a key fact you keep for months stays. A smudge on a page you consult often throws off your answers more. STEPQuant decides how carefully to recopy based on how long an error will live and where it lands.

The mechanism: rewriting state with the delta rule

The paper targets Gated DeltaNet (GDN) and Kimi Delta Attention (KDA) (§2.1). For one head, the state $S_t$ is a $d_k \times d_v$ matrix updated as follows.

In words: fade the old memory with the retention gate $D_t$, look up what the key $k_t$ currently retrieves, and write in only the difference from the correct value $v_t$. $\beta_t$ is the write strength, $q_t$ is the read query, and $y_t$ is the head output. GDN uses one scalar gate per head; KDA uses a separate gate per key channel.

The intuition: errors get carried forward

When you run with a quantized state, the gap $E_t$ from the true state evolves like this (§3.1, Proposition 1).

In words: today's error equals yesterday's error, shrunk a little, plus the fresh error $\varepsilon_t$ from tonight's recopy. Two forces do the shrinking: gate decay, and the delta-rule overwrite, which only trims the component along the current key. Errors pointing in directions later keys rarely hit depend almost entirely on the gate, so in heads whose gates sit near one, they survive for a long time.