#quantization

13 articles

01 ·Time Series·★ MEMBER·PAPER·7 min read Paper Explained: STEPQuant — When and Where Quantization Errors Matter in Delta-Rule Recurrent States Linear-attention states have a fixed size, yet under concurrent serving they can outgrow the model weights. Naive low-bit quantization lets errors pile up with every update. STEPQuant spends bits and scales according to how long an error survives and which row it lands on. Explained from scratch. 02 ·Inference & Serving·★ MEMBER·PAPER·6 min read Paper Explained: Disaggregated Quantization — Quantize LLM Prefill and Decode Differently A from-scratch explanation of disaggregated quantization: giving LLM prefill and decode their own number formats, weights and storage. Covers the prefillers that more than double 1-bit GGUF accuracy, SSD-streamed prefill (ODP), and where the method stops working. 03 ·Inference & Serving·★ MEMBER·PAPER·6 min read Paper Explained: DeepSeek-V4.1-Flash — How Far Can KV Cache Compression Go? DeepSeek-V4.1-Flash shrinks its global KV cache to 890 bytes per token. We unpack the four ideas behind it (the CED layer split, cross-layer KV sharing in CSA2, FP4 caching, and SWA Bounded Replay) from first principles. 04 ·Training & Alignment·FREE·PAPER·19 min read QLoRA — Fine-Tuning a 65B Model on One GPU A 65-billion-parameter model in 16-bit weights is 130 GB before you have trained anything. QLoRA gets that onto a single 48 GB card by storing the frozen weights in a 4-bit format shaped like the weights themselves, quantizing the quantization constants, and paging the optimizer to CPU — while gradients still flow through those 4-bit weights into a small 16-bit adapter. 05 ·Inference & Serving·FREE·PAPER·9 min read KV Cache Compression — Quantize, Evict, Share Your Way to Cheaper Long-Context Inference Raise the context length and your concurrency collapses. The culprit is the KV cache, and its size is one plain multiplication. That formula has only three terms you can touch, so compression has only three directions — lower the precision, throw tokens away, share heads. KIVI, H2O, SnapKV, GQA and attention sinks all line up on those three. 06 ·How Transformers Work·★ MEMBER·PAPER·11 min read Paper Explained: Why Gated DeltaNet Survives 4-Bit Quantization — NVFP4 W4A4 in a Hybrid 27B A walkthrough of the paper that tested — and overturned — the belief that the recurrent half of a hybrid LLM is too fragile for 4-bit quantization, and explained mechanistically why log-space gates and the delta rule erase quantization noise. 07 ·Distillation & Compression·★ MEMBER·PAPER·10 min read Distillation vs. Quantization vs. Pruning — Three Roads to a Smaller Model There are three roads to a smaller model: coarsen the number grid (quantization), remove weights outright (pruning), or rebuild the thing at a smaller size (distillation). A head-to-head comparison on compression ratio, accuracy, and implementation cost — and why, when you stack them, training-based methods go first and quantization goes last. 08 ·Inference & Serving·FREE·11 min read Cutting Inference Cost in Practice — What to Do First Quantization and distillation both work, but in most shops there are two or three levers left that cost you nothing in quality. Break the bill into four numbers, then reorder the work: cache, batch, shorten, compress. What each lever actually buys, and what breaks when you take them out of order. 09 ·Inference & Serving·FREE·8 min read LLM Quantization from Scratch — Why Losing Precision Doesn't Break It Cut a 16-bit weight down to 4 bits and the model keeps writing. Why doesn't it break? Starting from two numbers — scale and zero point — through the reason weights survive but activations resist (outliers), the fork between PTQ and QAT, what INT8 and INT4 actually do, and the single most common evaluation accident: signing off on perplexity alone. 10 ·Inference & Serving·FREE·8 min read LLM Quantization from Scratch — Why Losing Precision Doesn't Break It Cut a 16-bit weight down to 4 bits and the model keeps writing. Why doesn't it break? Starting from two numbers — scale and zero point — through the reason weights survive but activations resist (outliers), the fork between PTQ and QAT, what INT8 and INT4 actually do, and the single most common evaluation accident: signing off on perplexity alone. 11 ·Accelerators·FREE·9 min read How Numbers Are Represented — From FP32 to FP8 and INT4 Open up the sign, exponent and mantissa fields and one rule falls out: the exponent buys range, the mantissa buys precision. From there — why bfloat16 was invented, why FP8 ships in two flavours, and what integer quantization actually destroys. Ends with which format to pick for training versus inference, and how to notice degradation before your users do. 12 ·Accelerators·FREE·9 min read How Numbers Are Represented — From FP32 to FP8 and INT4 Open up the sign, exponent and mantissa fields and one rule falls out: the exponent buys range, the mantissa buys precision. From there — why bfloat16 was invented, why FP8 ships in two flavours, and what integer quantization actually destroys. Ends with which format to pick for training versus inference, and how to notice degradation before your users do. 13 ·Image Codecs·★ MEMBER·9 min read Why JPEG Degrades — The DCT and Quantization from Scratch When a photo shrinks to a tenth of its size, what exactly was thrown away? Colour conversion, 8×8 blocks, the DCT, the quantization table and the zigzag scan — followed by what the number in "quality 90" really is, and where blocking and mosquito noise are born. Ending with the decisions people who ship images actually make.