#batching
2 articles
01
·Inference & Serving·FREE·PAPER·10 min read
LLM Serving from Scratch — vLLM, Continuous Batching, and Not Letting the GPU Idle
Getting a model to run and getting it to handle a hundred users are different problems. Without touching the weights or changing a single output token, the order and grouping of requests can multiply what one GPU delivers. This walks through why — arithmetic intensity, continuous batching, PagedAttention — and why throughput and latency can never both win.
02
·Inference & Serving·★ MEMBER·11 min read
Cutting Inference Cost in Practice — What to Do First
Quantization and distillation both work, but in most shops there are two or three levers left that cost you nothing in quality. Break the bill into four numbers, then reorder the work: cache, batch, shorten, compress. What each lever actually buys, and what breaks when you take them out of order.