Learn › AI

Inference & Serving

KV cache, quantization, batching, serving

01 ·Inference & Serving·FREE·7 min read The KV Cache from Scratch — The Heart of Fast Inference An LLM emits one token at a time. Written naively, every single token costs a full recomputation of the whole sequence — a spectacular waste. The keys and values of past tokens never change again, and that one fact drops an entire order of magnitude. What you pay instead is memory, in an amount you can work out yourself, and that is why batch size and context length hit a ceiling. 02 ·Inference & Serving·FREE·8 min read LLM Quantization from Scratch — Why Losing Precision Doesn't Break It Cut a 16-bit weight down to 4 bits and the model keeps writing. Why doesn't it break? Starting from two numbers — scale and zero point — through the reason weights survive but activations resist (outliers), the fork between PTQ and QAT, what INT8 and INT4 actually do, and the single most common evaluation accident: signing off on perplexity alone. 03 ·Inference & Serving·FREE·PAPER·8 min read Speculative Decoding from Scratch — How a Tiny Draft Model Speeds Up an LLM Without Changing a Single Output Speculative decoding lets a small model draft several tokens ahead while a large model verifies them in one batch — provably identical outputs, 2–3x faster in practice. A from-scratch walkthrough of the mechanism and the intuition behind the acceptance rate α, straight from the original paper. 04 ·Inference & Serving·FREE·PAPER·9 min read KV Cache Compression — Quantize, Evict, Share Your Way to Cheaper Long-Context Inference Raise the context length and your concurrency collapses. The culprit is the KV cache, and its size is one plain multiplication. That formula has only three terms you can touch, so compression has only three directions — lower the precision, throw tokens away, share heads. KIVI, H2O, SnapKV, GQA and attention sinks all line up on those three. 05 ·Inference & Serving·FREE·PAPER·10 min read LLM Serving from Scratch — vLLM, Continuous Batching, and Not Letting the GPU Idle Getting a model to run and getting it to handle a hundred users are different problems. Without touching the weights or changing a single output token, the order and grouping of requests can multiply what one GPU delivers. This walks through why — arithmetic intensity, continuous batching, PagedAttention — and why throughput and latency can never both win. 06 ·Inference & Serving·FREE·PAPER·11 min read Structured Output and Constrained Decoding — How to Stop an LLM from Breaking Your JSON Instead of asking a model nicely to return JSON, you can drive the probability of every grammatically illegal token to exactly zero before it is ever sampled. A from-scratch walkthrough of constrained decoding — logit masks, vocabulary indexing, what function calling really does, and why syntax guarantees are not content guarantees. 07 ·Inference & Serving·FREE·PAPER·11 min read Surviving GPU Out-of-Memory — Every Cause, Every Fix `CUDA out of memory` reports only the allocation that happened to fail last, which is almost never the culprit. We count what actually occupies VRAM in five buckets, derive the sixteen-bytes-per-parameter fixed cost of training, and work through the fixes in order of least damage: gradient checkpointing, optimizer compression, offloading, KV cache limits, and fragmentation. 08 ·Inference & Serving·FREE·11 min read Cutting Inference Cost in Practice — What to Do First Quantization and distillation both work, but in most shops there are two or three levers left that cost you nothing in quality. Break the bill into four numbers, then reorder the work: cache, batch, shorten, compress. What each lever actually buys, and what breaks when you take them out of order. 09 ·Inference & Serving·FREE·10 min read Prompt Caching and Context Design — One Prefix Rule That Moves Your Bill by an Order of Magnitude Are you paying to have the same system prompt re-read on every single request? Prompt caching only works on exact prefixes — and that one rule decides what goes where in your context. Why a single timestamp at the top wipes out everything below it, and how misreading the TTL can make caching 25% more expensive than not caching at all. 10 ·Inference & Serving·FREE·PAPER·9 min read Paper Walkthrough: The Personalization Mirage — LLMs Invent a Version of You, and Their Self-Reports Point the Wrong Way How much of what a memory-equipped LLM 'knows' about you was never actually said? MirageBench judges 143,616 claims across 150 personas and 6 tasks, finding that all 12 tested models over-infer 35–49% of their claims — and that the models reporting the least over-inference tend to fabricate the most. 11 ·Inference & Serving·FREE·PAPER·8 min read Paper Walkthrough: DAPD — Breaking the Teacher's "Cheat-Sheet Illusion" in Distillation with Dual Anchors When you distill from a teacher that can see the reference solution, the student learns to act as if invisible answers were still there — the privilege illusion. DAPD (arXiv:2608.01735) traces this failure to information asymmetry and fixes it with two levels of anchoring. A walkthrough grounded strictly in the paper itself. 12 ·Inference & Serving·FREE·PAPER·8 min read Paper Walkthrough: CodeNib — A Multi-View Data System That Serves Repository Context to Coding Agents Coding agents grep their way through the same repository over and over. CodeNib (UC San Diego et al.) reframes this as a database problem — lexical, dense, and structural views over an immutable commit — and measures the whole lifecycle, caveats included. 13 ·Inference & Serving·FREE·PAPER·8 min read Paper explained: BDH-CQ — an AI that thinks without words. Recurrent memory plus latent reasoning resets ARC's cost frontier BDH-CQ (arXiv:2608.09888) never verbalizes its intermediate reasoning: demonstrations are written into a recurrent memory, and the answer is computed by iterating in a high-dimensional latent space. A 150M-parameter model reaches 29.5% pass@2 on ARC-AGI-1 at $0.0007 per task, breaking the reported cost–accuracy Pareto frontier. We explain the paper using only its own text as evidence. 14 ·Inference & Serving·FREE·PAPER·9 min read Paper Deep-Dive: AgentOPSD — Finding the Turn That Won the Game with Recursive Bayesian Belief Updates When a long multi-turn episode only tells you 'success' or 'failure' at the very end, which turn actually mattered? AgentOPSD reinterprets self-distillation gaps as Bayesian evidence and recursively updates a success belief in log-odds space. We walk through the mechanism, the 89.1% ALFWorld result, the ablations, and the limitations — all from the paper itself. 15 ·Inference & Serving·FREE·PAPER·14 min read Paper Walkthrough: No Gold Answers, No Stronger Teacher — How u-OPSD Distills From Its Own Majority Vote u-OPSD (arXiv:2608.06296) replaces the gold solution in on-policy self-distillation with the model's own majority vote, then corrects only the rollouts that disagree with it. A from-scratch walkthrough grounded solely in the paper. 16 ·Inference & Serving·★ MEMBER·PAPER·15 min read Paper Explained: Agentic ESOpt — Drop Backprop, Jiggle the Weights, and Train Long-Horizon LLM Agents No gradients at all: spawn G perturbed copies of the model, run them in the environment, and nudge the weights toward the ones that scored well. That buys full-parameter updates of a 27B agent at inference-level memory (8.41GB) — and a 12.50-point lead over GRPO on 15-move Sudoku. A ground-up walkthrough of the NUS-led paper. 17 ·Inference & Serving·★ MEMBER·PAPER·8 min read Paper Walkthrough — WarpSAC: When RL's Safety Rails Become Handcuffs Once GPU-parallel simulators flood the replay buffer, SAC's normalization and clipped double-Q flip from helpful to restrictive. The paper isolates three design axes and prescribes removing stabilizers — not stacking them — when data are abundant. 18 ·Inference & Serving·★ MEMBER·PAPER·11 min read TTPO Explained: Training a Model Mid-Exam, With No Answer Key Majority-vote pseudo-labels are wrong on roughly 85% of competition math prompts. So why does training still work? A ground-up walkthrough of TTPO's asymmetric design: distill the rollouts that agree, penalize the ones that disagree. 19 ·Inference & Serving·★ MEMBER·PAPER·13 min read Paper explained: Self-OPD — an image generator that distills itself, with no teacher No teacher model at all: at every denoising step the student spawns K copies of itself, scores them, and pulls toward the good branches while pushing away from the bad ones. A ground-up walkthrough of Self-OPD, an alignment method for flow matching image generators. 20 ·Inference & Serving·★ MEMBER·PAPER·11 min read Paper walkthrough: CyberFactory — turning wild CVEs into runnable training problems An open-source pipeline that converts real CVEs into executable, verifiable tasks and uses a reusable vulnerability-analysis skill to synthesize teacher trajectories the student then internalizes. 58.1% Pass@1 on CyberGym. 21 ·Inference & Serving·★ MEMBER·PAPER·12 min read Paper Walkthrough: DART-SD — Training Tool-Calling Agents Without Flattening the Diamond When a multi-turn tool-calling task has order-independent sub-goals, the set of correct solutions spreads out into a diamond lattice. This walkthrough explains why whole-trajectory imitation crushes that structure, and how locating the first point where a rollout leaves recoverable territory — and supervising only what comes after it — changes the picture. 22 ·Inference & Serving·★ MEMBER·PAPER·16 min read SMELT explained (arXiv:2609.01343) — compute-matched looping Does running the same layers twice help once the budget is matched? SMELT holds per-token FLOPs, total parameters and KV cache within about 4%, loops the middle half twice, and reports 6.8-18.0% training-FLOPs savings on the compute-optimal frontier. 23 ·Inference & Serving·★ MEMBER·PAPER·9 min read Paper Walkthrough: Normalized Low-Rank Adaptation — Why Normalizing LoRA's Entry Matrix Works Rescaling LoRA's down-projection so every column has unit length improves convergence, stability and forgetting resistance at zero extra cost. A ground-up reading of Normalized Low-Rank Adaptation (NoRA) through the lens of a hidden preconditioner. 24 ·Inference & Serving·★ MEMBER·PAPER·14 min read Paper Walkthrough: From Production Traffic to Post-Training — Folding 200 Internal Apps Into One Self-Hosted LLM How a company forced to self-host by data-residency rules collapsed a sprawling model fleet into one checkpoint: classify production failures by hand, train one GRPO expert per weak axis, then merge the weights with SLERP. Includes three real reward-hacking failures. 25 ·Inference & Serving·★ MEMBER·PAPER·11 min read Paper Explained: Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement A third to a half of the teacher's token-level grades are wrong, yet the student improves just as fast either way. This paper traces on-policy distillation's gains not to imitating a teacher but to suppressing the student's own low-probability tokens — and drops the teacher entirely. 26 ·Inference & Serving·★ MEMBER·PAPER·12 min read Paper Explained — LatentPress: Feeding Compressed Context Straight to a Frozen LLM, Neither as Text Nor as Pixels A method that writes long conversation histories and documents into continuous memory tokens a frozen LLM reads directly — no text summary, no rendered image. On LongMemEval it beats uncompressed evidence at 7.70x compression. Explained from first principles. 27 ·Inference & Serving·★ MEMBER·PAPER·9 min read Paper Walkthrough: Language Models Can Control Their Own Attention During long-context decoding, a model re-reads its entire KV cache at every step. Declarative Attention has the model announce, inside its own chain-of-thought, where it will look next, and the inference engine builds an attention mask from that text. A walkthrough of the paper that cut attended tokens by 52.0% and 31.1% on off-the-shelf models, zero-shot. 28 ·Inference & Serving·★ MEMBER·PAPER·10 min read Paper Explained: Compile by Training — Turning a Natural-Language Spec into a Function That Runs Locally "Sort my email into urgent and later" — a spec that vague, turned into a small function you can run locally after about a minute of training. A ground-up walkthrough of Compile by Training: analogy, mechanism, equations, measured numbers. 29 ·Inference & Serving·★ MEMBER·PAPER·11 min read Paper Walkthrough: One Training Example Keeps On-Policy Distillation Improving for Hundreds of Steps Trained on a single query, on-policy distillation still improves for hundreds of steps and recovers most of full-data OPD's gain. The paper explains this with state coverage and absorption rate, and concludes OPD is data-overfed but algorithm-starved. 30 ·Inference & Serving·★ MEMBER·PAPER·12 min read Paper Walkthrough: Random Attention — Throwing KV Cache Entries Away at Random Works Just as Well The importance score that decides what leaves the KV cache turns out to buy almost nothing. Keep the prompt, evict uniformly at random inside each head, and you match the strongest prior evictor while serving 32-43% more tokens per second in vLLM. A walkthrough of the measurements across four models and six tasks, and the two controlled experiments that explain them. 31 ·Inference & Serving·★ MEMBER·PAPER·11 min read Paper Walkthrough: On-Policy Reverse Distillation — Borrowing a Weak Teacher's Direction, Not Its Answers OPRD takes the policy change a weak teacher acquired during post-training and uses it only as a direction to amplify the student's own policy gradient — never as an optimization target. A ground-up walkthrough. 32 ·Inference & Serving·★ MEMBER·PAPER·12 min read Paper walkthrough: BeaconKV — evicting KV cache by predicting when a reasoning model will re-read its own plan Deep into a long chain of thought, a reasoning model occasionally looks back at the plan it wrote at the start. The queries that do this cluster into a handful of groups in embedding space — so keeping a few dozen cluster representatives (beacons) is enough to predict which KV pairs will be revisited. A training-free method with up to 5.8× less memory and over 4.3× more throughput, read through its equations and code. 33 ·Inference & Serving·★ MEMBER·PAPER·10 min read Paper Walkthrough: Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training — training the draft alongside a 122B policy at 256K tokens Rollout generation dominates RL post-training time. NVIDIA speeds it up with speculative decoding and keeps training the draft model alongside the policy — and shows how to get past the two walls that context parallelism and pipeline parallelism put in the way. A from-scratch walkthrough grounded in the paper text. 34 ·Inference & Serving·★ MEMBER·PAPER·7 min read Paper Explained: Steering Geometry — Do LLM Steering Vectors Trace the Map of Human Values? Activation steering nudges an LLM's hidden states at inference time to change its behavior. This paper asks whether those nudges follow the circular structure of human values, using a 20-value, ~26K-sample benchmark. Explained from zero. 35 ·Inference & Serving·★ MEMBER·PAPER·10 min read Paper Explained: FreeFlow — Optical Flow with a Hierarchical Transformer and No Flow-Specific Parts FreeFlow drops correlation volumes, warping, and iterative refinement, and reaches the top of Sintel, KITTI-2015, and Spring using only window, shifted-window, and low-resolution global attention. We walk through it from zero. 36 ·Inference & Serving·★ MEMBER·PAPER·7 min read Paper Explained: An Open Recipe for IMO Gold — How Open Models Reached Olympiad Gold with Generate, Verify, Refine NVIDIA's open recipe uses three Nemotron 3 Ultra checkpoints to generate, verify, and refine proofs, scoring 30/42 (gold-medal level) at IMO 2026. We explain it from scratch as a test-time-compute design. 37 ·Inference & Serving·★ MEMBER·PAPER·7 min read Paper Explained: When EOS Tokens Disagree — Why On-Policy Distillation Students Forget How to Stop In on-policy distillation, student responses can grow until they hit the generation budget. This paper (arXiv:2609.20511) traces a major cause to student and teacher marking 'the end' with different special tokens. We walk through it from scratch, using only the paper itself. 38 ·Inference & Serving·★ MEMBER·PAPER·8 min read Paper Explained: Negative Self-Distillation — Learning to Reason by Steering Away From Your Worst Self Self-distillation from ground-truth solutions (OPSD) can strip a reasoning model of its habit of pausing and double-checking. NSD has the model play a 'careless reasoner' and trains only to move away from that bad teacher — no labels required. A from-scratch walkthrough. 39 ·Inference & Serving·★ MEMBER·PAPER·6 min read Paper Explained: IntBMoE — Every Expert Contributes, Yet Each Token Runs Only a Few Blocks In a Mixture-of-Experts layer, how many experts inform a token, how many get computed, and how many weight sets must be built have always been tied together. IntBMoE (arXiv:2609.21346) cuts those ties by composing experts densely and executing blocks sparsely. Explained from scratch, using only the paper itself. 40 ·Inference & Serving·★ MEMBER·PAPER·6 min read Paper Explained: DeepSeek-V4.1-Flash — How Far Can KV Cache Compression Go? DeepSeek-V4.1-Flash shrinks its global KV cache to 890 bytes per token. We unpack the four ideas behind it (the CED layer split, cross-layer KV sharing in CSA2, FP4 caching, and SWA Bounded Replay) from first principles. 41 ·Inference & Serving·★ MEMBER·PAPER·6 min read Paper Explained: JEV-as-a-Judge — Accept When Confident, Escalate When Unsure A cheap judge that returns only a verdict and label probabilities handles the first pass, and only its low-confidence calls go to a strong LLM. This walkthrough covers the paper's cost measurements, where the two-stage setup works, and where it breaks. 42 ·Inference & Serving·★ MEMBER·PAPER·8 min read Paper Explained: RewardVerse — A Video Reward Model That Writes the Rubric Before It Scores (RGPO) Video reward models tend to squeeze their scores into a narrow band, and those scores move when the wording changes. The paper calls this scalar drift. RewardVerse first generates an evaluation rubric, then reads a continuous score for each item from token probabilities. Its two-stage RGPO training uses only 30 preference pairs per dimension. 43 ·Inference & Serving·★ MEMBER·PAPER·6 min read Paper Explained: Disaggregated Quantization — Quantize LLM Prefill and Decode Differently A from-scratch explanation of disaggregated quantization: giving LLM prefill and decode their own number formats, weights and storage. Covers the prefillers that more than double 1-bit GGUF accuracy, SSD-streamed prefill (ODP), and where the method stops working. 44 ·Inference & Serving·★ MEMBER·PAPER·9 min read Paper Explained: Post-Training Leaves Behavioral Shadows on Unrelated Decisions — capability transfer through one unrelated word Ask a code-tuned model to pick between 'jacket' and 'tie' a few thousand times, and another model can inherit part of its coding gain. A from-scratch walkthrough of Active Taskless Distillation (ATD): the math, the controls, and the limits. 45 ·Inference & Serving·★ MEMBER·PAPER·8 min read Paper Explained: Beyond Teacher Assignment — Why Choosing the Right Teacher Isn't Enough When You Distill Many Specialists into One Model When MOPD distills math, code and instruction-following specialists into one student, the instruction-following teacher shouts so loudly that math barely transfers. DN-MOPD measures each teacher's feedback spread and evens out the volume, with a one-line change.