Paper Explained: Disaggregated Quantization — Quantize LLM Prefill and Decode Differently

A from-scratch explanation of disaggregated quantization: giving LLM prefill and decode their own number formats, weights and storage. Covers the prefillers that more than double 1-bit GGUF accuracy, SSD-streamed prefill (ODP), and where the method stops working.

Disaggregated Quantization: Specializing LLM Prefill and Decode

What this paper claims

The paper is "Disaggregated Quantization: Specializing LLM Prefill and Decode" by Panferov et al. (NVIDIA and ISTA), posted on September 22, 2026.

Here is the abstract in plain terms. LLM inference runs in two phases. Prefill reads the prompt, and decode generates the answer one token at a time. Each phase gains from quantization in a different way: prefill speeds up when the arithmetic itself is low-precision, while decode speeds up when compact weights cut memory traffic. The authors propose "disaggregated quantization" (DQ), which specializes the compute formats, the weights and the storage placement for each phase. Turning off activation quantization only during decode improves accuracy at no extra cost. Training a separate set of prefill weights makes prefill faster than weight-only inference, and it matches or beats that baseline's accuracy at 2–3 bits. On released Qwen3.8-27B GGUF checkpoints, adding a trained prefill model without changing the decode checkpoint raises 1-bit accuracy by 32.5 points on MMLU-Pro and 35.3 points on MMMU-Pro. The extra weights are streamed from SSD (ODP), and in llama.cpp this delivers a 1.78× time-to-first-token (TTFT) speedup at an 8K prompt.

For background, see Quantization from scratch, and KV cache explained for the cache that prefill builds.

An analogy: the reader and the writer

Picture someone who has to read a thick dossier and write a short memo about it. A reader goes through the whole dossier in one pass and condenses it onto sticky notes (the KV cache). Because they process hundreds of pages at once, raw speed (compute) is what limits them. A writer then produces the memo one character at a time, checking the notes and pulling a heavy dictionary (the weights) off the shelf for every character. Their bottleneck is carrying the dictionary around (memory traffic). Quantization has traditionally given both workers the same tools. This paper argues that each one should get their own.

Intuition: why the two phases differ

Per §1.1, multiplying an $H\times H$ weight matrix by an input of $T$ tokens moves roughly $H^2+TH$ values from memory and performs roughly $TH^2$ operations. Dividing compute by traffic gives the following (writing it as one ratio is our own step):

In words: the more tokens you process at once, the more arithmetic each weight pays for per trip from memory. This number says how many times each weight gets reused per trip from memory. In decode, $T=1$, so $I\approx 1$ and traffic is the bottleneck, which is why compressing the weights speeds decode up. In prefill, $T$ runs into the thousands, so arithmetic becomes the bottleneck and you need a format like NVFP4 that the hardware can compute in natively. Running an exotic 2–3-bit encoding through fast prefill math means re-quantizing the weights to NVFP4 and quantizing the activations too. A single format can't serve both phases well.