LLM Serving from Scratch — vLLM, Continuous Batching, and Not Letting the GPU Idle

Getting a model to run and getting it to handle a hundred users are different problems. Without touching the weights or changing a single output token, the order and grouping of requests can multiply what one GPU delivers. This walks through why — arithmetic intensity, continuous batching, PagedAttention — and why throughput and latency can never both win.

Efficient Memory Management for Large Language Model Serving with PagedAttention

"It runs" and "it holds up" are different problems

You start a model on a GPU, send it a question, and get an answer back. It runs. Then a hundred people connect to that same server at once, replies crawl, and eventually the process dies with an out-of-memory error. Running a model once and absorbing a stream of arriving requests are two different jobs.

The layer that does the second job is called serving. vLLM, TensorRT-LLM, TGI, and SGLang all live there. The interesting part is that this layer doesn't touch the model at all. Same weights, same arithmetic, same output tokens. All it decides is when, in what order, and grouped with what work reaches the GPU. That alone changes how much a single GPU can absorb by a large multiple.

A metaphor: the cafeteria that never refuses to seat you

Picture a cafeteria with eight seats that makes people wait at the door until eight have gathered, seats them all at once, and admits nobody else until every one of the eight has finished. Someone who eats in three minutes leaves a seat empty until the person working through a full course is done. That is static batching.

Now picture the opposite: the moment a seat opens, the next person sits down. Meals no longer start and finish in lockstep, but almost no seat sits idle. That is continuous batching.

With LLMs the gap between these two is extreme. One reply ends after twelve tokens; another runs to eight hundred. Worse, you cannot know the output length in advance. Any design that waits for a group to finish together is, by construction, a design that leaves seats empty.

There are two kinds of work: prefill and decode

LLM inference splits into two phases with completely different characters.

Prefill reads the whole prompt at once. A thousand prompt tokens can be processed simultaneously, so the GPU's arithmetic units are packed full. This phase is limited by compute.

Decode emits the answer one token at a time. A single step handles exactly one token — and yet it has to read every weight in the model to do it. Tens of gigabytes read in, a trivial amount of multiplication performed. This phase is limited by memory bandwidth, and the arithmetic units sit mostly idle.

That asymmetry is the origin of everything else here. The mechanism for reusing past keys and values belongs to The KV Cache from Scratch; the only thing you need to carry forward is one sentence. During decode, the GPU is not busy computing — it is waiting on weight reads.

Why grouping requests is nearly free

There is exactly one way to fill those idle arithmetic units: while the weights are being read anyway, finish another request's math with them too.

The number that captures this is arithmetic intensity — how many operations you perform per byte you read.

The smaller $I$ is, the more you are reading rather than computing, which is another way of saying you are pinned to memory bandwidth.

Apply it to one decode step. A model with $P$ parameters stored in fp16 (two bytes each) requires reading $2P$ bytes; processing $B$ requests together costs roughly $2PB$ operations.

$$I \approx \frac{2PB}{2P} = B$$

Arithmetic intensity is just the batch size. Going from $B = 1$ to $B = 32$ does not change how many bytes you read. Read the weights once, reuse them for all thirty-two. Which means thirty-two requests take about as long as one. When a serving system claims a several-fold speedup, this is almost always what it means.

So why not push $B$ arbitrarily high? Because each in-flight request holds KV cache in memory. For a model with 32 layers, 8 KV heads, head dimension 128, in fp16, one token costs $2 \times 32 \times 8 \times 128 \times 2 = 131{,}072$ bytes — 128 KiB. A single 2,000-token conversation is 256 MiB; a hundred of them at once is 25 GiB. The ceiling on batch size is set by memory, not by compute.

Continuous batching: rebuild the batch every step

The first fix is continuous batching. Its ancestor is a research system called Orca and its iteration-level scheduling: rebuild the batch not per request but every time you advance by one token (TensorRT-LLM calls the same idea in-flight batching). Drop the requests that finished, pull in the head of the waiting queue. Output lengths can be wildly uneven and the batch still stays close to full.

An open seat you can't sit in: fragmentation and PagedAttention

Except continuous batching alone doesn't deliver. The seat that opened up is a logical seat — and there may be nowhere in memory to actually put the new occupant.

Implement the KV cache naively and you reserve, per request, a contiguous region sized for the maximum possible length. Reserve for 4,096 tokens, finish at 300, and the rest stays locked up (internal fragmentation). Let lengths vary and gaps open between regions, so that memory is free in total but no single request fits (external fragmentation). The vLLM paper reports that in existing systems, less than half of the reserved KV region actually held token state. Waste half the memory and you fit half the batch; halve the batch and you halve the arithmetic intensity from equation (1). Halving arithmetic intensity halves throughput.

vLLM's answer was to import a solution operating systems have used for forty years: give up on contiguity. Instead of reserving ten consecutive library shelves under one number, scatter the material across whatever shelves are free and keep an index to find it. Cut the KV cache into fixed-size blocks of a set number of tokens (--block-size) and keep a table mapping logical block number to physical block number. The attention kernel walks that table, gathering K and V as it computes. This is PagedAttention.

Now the only waste is the remainder inside the last block. More requests fit at once → batches grow → arithmetic intensity rises. The paper reports 2–4× the throughput at the same latency level as the systems it compared against. There's a bonus, too: physical blocks can be shared. A hundred requests beginning with the same system prompt store that prefix once and copy only on write. That's what --enable-prefix-caching does.

Prefill grows with the square of the prompt

There is one more asymmetry you will hit the moment you start measuring. Decode costs roughly constant time per token, but the attention part of prefill grows with the square of the prompt length. Double the prompt, quadruple the work; ten times the prompt, a hundred times the work.

This lands directly on the time to the first visible character — and worse, while a long prefill is running, every other request currently decoding stalls. --enable-chunked-prefill splits long prefills into pieces and interleaves them between decode steps precisely to prevent that stall.

The skeleton of a scheduler

Written out, the core of all this is surprisingly short.

while True:
    # 1) admit from the queue while free blocks remain
    while waiting and can_allocate(waiting[0]):
        running.append(admit(waiting.popleft()))

    # 2) advance every live request by one step (one fused forward pass)
    logits = model.step(running)          # prefill and decode may be mixed
    for req, tok in zip(running, sample(logits)):
        req.append(tok)

    # 3) retire what finished, return its blocks
    for req in finished(running):
        free_blocks(req)
        running.remove(req)

    # 4) out of memory? don't crash — evict
    while out_of_blocks():
        preempt(running[-1])              # swap to CPU, or drop and recompute later

Step 4 is the one that matters operationally. When memory runs out the server does not fall over; it temporarily evicts requests that arrived later (preemption). Evicted requests resume afterwards, so nothing shows up as an error. Instead it surfaces as particular requests freezing mid-stream for an unusually long time. If you don't know preemption exists, this looks like an inexplicable latency anomaly.

Throughput and latency cannot both improve

Grow the batch and each step takes longer, and the queue behind it lengthens too. Total work processed goes up, but each individual user waits more. The one-line statement of that relationship is Little's Law.

$L$ is the average number of requests in the system, $\lambda$ the arrival rate per second, $W$ the average time a request spends from entry to exit. Read it backwards: the $L$ you can hold at once is capped by KV memory, so if you want to accept a higher arrival rate, you have to shorten how long each request stays. If you can't shorten it, you have to make people wait. Push load higher and throughput flattens at some point, and past that point only the queue grows. Operating beyond that saturation point buys you slowness and not one extra unit of work.

How to read the measurements

Which is why "fast" can never be a single number. Watch at least three. TTFT (time to first token — determined by queueing plus prefill), TPOT/ITL (per-token time after the first — determined by how crowded decoding is), and throughput (system-wide output tokens per second — the denominator you divide your GPU bill by).

Four ways to get tripped up:

1. Look at p95/p99, not the mean. Continuous batching improves the average while widening the spread. The mean hides the few percent of users frozen by preemption.
2. Check the subject of "tokens per second." Per request, or summed across the system? The same configuration can differ by more than 10×, and most contradictions between comparison articles come from exactly this.
3. Match the input/output length distribution to production. Benchmark with uniform-length synthetic data and the prefill-to-decode ratio drifts away from reality, sometimes far enough to invert the conclusion.
4. Throw away the warmup, and suspect your client. The first few requests are slow for reasons like CUDA graph capture — and more than once, the thing that was slow turned out to be the measurement script.

vLLM ships benchmarks/benchmark_serving.py, which drives load at a specified arrival rate and reports all of these together. Raise the arrival rate in steps and find where p99 TTFT crosses your acceptable line — that is the saturation point, measured rather than guessed. Counting only the requests that met an SLO (say, TTFT under 1 second and TPOT under 50 ms) gives an effective throughput called goodput, which is increasingly the number people compare on.

How this is used in practice

Who touches it, and when. Inference-platform, MLOps, and platform engineers — when exposing a model as an internal API, when sizing how many GPUs to buy, and when triaging "things have felt slow lately."

The knobs you actually turn (vLLM).

• --gpu-memory-utilization: the fraction of memory beyond the weights given to the KV pool. Raising it grows batches and shrinks your margin before OOM.
• --max-model-len: set it well above real demand and the per-request reservation ceiling balloons, cutting concurrency. "Max, just in case" is the single most common misconfiguration.
• --max-num-seqs / --max-num-batched-tokens: concurrency and per-step token ceilings. These are the throttle you use to bound latency.
• --enable-chunked-prefill / --enable-prefix-caching: the first stops long prompts from stalling decode; the second pays off in proportion to how much of a shared prefix your requests have.
• --tensor-parallel-size: for when the model doesn't fit on one card. It adds communication, so don't raise it when the model already fits.
• --quantization / --kv-cache-dtype: shrinking weights or KV widens the KV pool, which grows batches. The mechanics are in Quantization from Scratch.

The numbers to watch. vLLM exposes Prometheus metrics at /metrics. Five of them will resolve most incidents: vllm:num_requests_running, vllm:num_requests_waiting, vllm:gpu_cache_usage_perc, vllm:time_to_first_token_seconds, vllm:time_per_output_token_seconds. If waiting is piling up while cache usage sits near 100%, you are genuinely out of capacity. If waiting grows while cache usage stays low, you have throttled yourself artificially with something like max-num-seqs.

Pitfalls that turn into incidents.

• Concluding "it's fast" from a benchmark that sends one request at a time. Production sends them concurrently, and the numbers are not comparable.
• Setting gpu-memory-utilization too low out of fear of OOM, starving the KV pool and triggering constant preemption. The classic signature: the mean looks fine and only p99 spikes.
• Co-locating a monitoring agent or another process on the inference GPU, producing intermittent spikes.

How it gets asked. "Only TTFT regressed. Where do you look?" The reasoning goes to queue depth (num_requests_waiting) and prefill token counts. Did prompts get longer, or did the arrival rate rise? The first calls for chunking; the second is a capacity problem. The fixes have nothing in common, so distinguishing them is the whole answer.

There is also a different line of attack that breaks the sequential nature of decode itself. Speculative decoding advances several tokens per step without changing the output, and it pays off most exactly when batches are small and the arithmetic units are idle. Same goal, different way of filling them.

Summary

• Decode is memory-bandwidth bound. Grouping requests raises arithmetic intensity, letting one weight read serve many requests.
• Continuous batching rebuilds the batch every step; PagedAttention eliminates fragmentation so more requests fit in the batch at all.
• Throughput and latency are two ends of one dial. Past the saturation point you get slower without processing more.
• Measure TTFT, TPOT, and throughput — at p99 rather than the mean, with production-like length distributions and arrival rates.