Cutting Inference Cost in Practice — What to Do First

Quantization and distillation both work, but in most shops there are two or three levers left that cost you nothing in quality. Break the bill into four numbers, then reorder the work: cache, batch, shorten, compress. What each lever actually buys, and what breaks when you take them out of order.

Before you swap the bulbs, turn off the empty rooms

When your electricity bill triples, the first move is not replacing every bulb in the house with an LED. First you find out which rooms are burning it, then you switch off the lights nobody is using. Buying new fixtures comes last. Do it in the other order and you end up spending money and effort while the bill barely moves.

Inference cost has exactly this shape. "Let's quantize" and "let's distill into a smaller model" are the eye-catching moves, but in most deployments there are two or three levers ahead of them that work better and cost nothing in quality. This article is not a tour of the techniques themselves. It is about what order to pull them in, and what each one costs you on the way out.

The bill is made of four numbers

Whether you own the GPUs or call somebody's API, the structure of the payment is the same: a unit price for compute, times the time you occupied it. A month of it looks like this.

$C$ is the monthly cost, $p$ the price per GPU-hour, $N$ the number of requests in the month, $T_p$ the time to read one prompt (the prefill), $L$ the average number of output tokens, $T_d$ the time to emit one token, and $B$ how many requests you manage to process together — the effective batch size.

Stripped of the symbols, it is a sentence which says: price × (how many arrive × how much work each one is) ÷ how many you clear at once. That is the whole thing.

What makes the formula useful is that it immediately tells you there are only four kinds of moves. Shrink $N$ (don't call), shrink $L$ (make it shorter), shrink $T_d$ (make each token lighter), grow $B$ (batch them). Every cost-cutting technique in circulation lands on one of those. Which also means that when someone brings you a proposal, you can ask "which letter does this move?" and judge it on the spot.

And the axis that sets the order is not how big the win is. It is whether you are gambling with quality. Growing $B$ does not change a single character of the output; most ways of shrinking $T_d$ change the output itself. When two moves buy the same thing, take the one that isn't a gamble. That is the skeleton of the whole plan.

Move 0: Find out where the money is going

You cannot skip the measurement. Three things to look at.

The mix of requests. Break traffic down by endpoint and by feature, and rank it by GPU-seconds, not request count. A feature that looks trivial by volume often turns out to be half the bill because its outputs are long.

The distribution of input and output lengths. Report the median and p95, not the mean. Averages hide the picture where a small number of enormous requests are sitting on the GPU for a long time. The $L$ in the formula is a mean, but what usually hurts you lives in the tail.

Duplicate calls. How often does the same input arrive twice? Are retries or agent loops calling the model two or three times for one user action? It is not unusual for this to be tens of percent of traffic.

Move 1: Don't call the model at all

The cheapest inference is the one you never run, and by definition it costs nothing in quality. There are three tiers of it.

Exact-match cache. Normalize the input, and if you've seen it, return the stored output. It's a few dozen lines of code, and it pays off well for anything used in a FAQ-ish way. One condition: set temperature to 0 so generation is deterministic. Bolt a cache onto a sampling setup and you permanently freeze the artifact where the same question gets different answers depending on who asked first.

Prefix cache. If a hundred requests begin with the same system prompt, the computation for that shared span can be reused. The mechanism for keeping past keys and values around is the one covered in Understanding the KV Cache from Scratch; this is that mechanism shared across requests. In vLLM it is one flag, --enable-prefix-caching. The output does not change by a single bit, and for setups with a long system prompt the prefill time simply disappears. If you change one setting today, change this one.

Semantic cache. Embed the input, and if a close-enough past question exists, return its answer. Hit rates go up — and this is the tier where you start gambling.

The failure mode is predictable. "Tomorrow's weather in Tokyo" and "tomorrow's weather in Osaka" sit very near each other in embedding space. So do "does this contract auto-renew?" and "does this contract not auto-renew?" Swap a proper noun, a number, or a negation and the meaning inverts while the distance barely moves. The workable compromise is a tight threshold plus a rule that excludes requests containing proper nouns or figures from the cache entirely. Your hit rate drops, but confidently returning the wrong answer costs far more than a miss.

Move 2: Batch — the biggest win that costs no quality

After caching comes $B$, and this is where most deployments are leaving the most on the table.

The reason is in the shape of the work. To emit one token, the model must read all of its weights out of memory. Tens of gigabytes read, a trivial amount of arithmetic done. The GPU's compute units spend that step waiting. If you let another request ride along during the same read, the number of bytes read does not go up. Thirty-two requests finish in about the time one would have. That's essentially the whole story behind "we put a serving stack in and got an N× speedup."

So if you are still on static batching — waiting until a fixed number of requests accumulate, then running them together — moving to a stack with continuous batching can more than double throughput on its own. The weights don't change and the outputs don't change, so you don't even have to re-run your evaluations.

There are two side effects. The first is a trade against latency: a bigger batch makes each step longer and the queue behind it longer. Total throughput rises while the experience of any one user gets slower. The second is variance: when memory gets tight, running requests get temporarily evicted, so a particular user stalls mid-answer. This surfaces as complaints arriving while your average latency chart looks better than ever. Watch p95 and p99.

The knobs come down to three: the share of memory given to the KV pool (--gpu-memory-utilization), the maximum context length that determines the per-request reservation (--max-model-len), and the cap on concurrency (--max-num-seqs). The second one is the most common misconfiguration there is: setting it to the maximum "just in case." If real traffic needs 2,000 tokens and you reserve for 128,000, per-request reservations balloon, concurrency falls, and $B$ drops with it.

Move 3: Make it shorter

$L$ maps to cost about as directly as anything does. Half the output, half the decode time.

On the output side. Put a realistic ceiling on max_tokens. Ask for brevity in the instruction. Force structured output such as JSON so the preamble and the hedging disappear. The side effect is unambiguous: output truncated at the ceiling is broken JSON. If you trim, budget for parse-failure retries — which are themselves cost.

On the input side. Pack three RAG chunks instead of ten, cut few-shot examples, replace conversation history with a summary. Input trimming attacks $T_p$, and the interesting part is that it isn't linear. As the prompt grows, the bulk of the model's compute grows in proportion, but the attention part grows with the square of the length. That's why shaving a long prompt slightly can visibly shorten the time to the first character.

The side effect here arrives quietly. Delete a few-shot example and overall accuracy barely moves, while the one pattern that example was covering — an unusual format, a particular language — falls off a cliff. Aggregate metrics will not show you this. Before you cut, build an evaluation set that explicitly names the cases the thing you're cutting was protecting.

Move 4: Make it smaller

Now we are in $T_d$ territory, and now you are gambling.

Quantization is the first candidate. Drop weights from fp16 to INT4 and you read a quarter as many bytes; since decode is spent waiting on bandwidth, fewer bytes read converts straight into speed. The procedure and its traps are in LLM Quantization from Scratch; from a prioritization standpoint only one thing matters. Do not ship on perplexity alone. Perplexity is an average over all tokens, so an increase in rare failures gets diluted into invisibility. What actually breaks first is instruction-following over long contexts, code generation, and the well-formedness of tool-call arguments — the kind of work where one miss ruins the whole response.

KV cache quantization should be treated as a separate decision. It coarsens the context rather than the weights, so it pays off on long conversations, but the damage shows up as attention pointing at the wrong place. Approach it more carefully than weight quantization.

Distillation and swapping in a smaller model give the largest win per unit — and carry the largest effort and the least reversibility. Collect teacher outputs, run training, build an evaluation, then redo all of it every time the production input distribution shifts. That is why it goes last. Before you get there, routing is worth trying: send the hard requests to the big model and let a small one handle the rest. Estimate both the cost of the router itself and the cost of the calls it misroutes before you commit.

Move 5: Be clever — but only when there's room

Speculative decoding has an unusual property. A small model drafts several tokens ahead and the large model verifies them in one pass. As long as you use the standard verification procedure, the output distribution is identical to the original model's. It is one of the rare moves that buys speed without a quality gamble. The details are in Speculative Decoding.

Even so, don't put it at the top of the list. Speculative decoding works by handing verification to compute units that were idle. If batching has already filled them, there is nothing left to fill. In internal or low-traffic environments — small $B$, where single-request latency is the complaint — it wins big. On a busy shared server the win evaporates. It is the clearest example of a lever whose rank depends on your situation.

What breaks when you take them out of order

The same set of moves produces a different outcome depending on sequence.

Quantize first and the bill moves less than the benchmark suggested, because utilization was the actual problem: idle GPU time stays idle no matter how light the model is. Meanwhile you've already spent your quality budget, and when batching later changes $B$, you get to measure the quantization effect all over again.

Cache last and you spent your tuning effort on requests you could have deleted; the moment the cache lands, your assumptions collapse.

Do everything at once and you cannot tell which change cost you the quality regression. Freeze an evaluation set, add one lever at a time, measure after each. It looks like the slow road and it is the fast one.

How this shows up on the job

Who touches it, and when. Inference-platform and MLOps engineers, and the backend owners of any product with an AI feature, reach for this when the GPU budget blows out, when growth forces the per-request cost down, or when leadership asks how much cheaper the same quality can get.

What you actually touch. On the server: --enable-prefix-caching, --gpu-memory-utilization, --max-model-len, --max-num-seqs, --enable-chunked-prefill. On the application side: temperature=0 plus a normalized cache key, max_tokens, and the number of RAG chunks retrieved. For measurement: TTFT (time to first token), TPOT (time per output token thereafter), system-wide output tokens per second, and goodput — throughput counting only the requests that met your SLO. For evaluation: a fixed set built from production logs, re-run identically after every change.

Traps that turn into incidents.

• Reporting cache hit rate as cost reduction. The questions that land in a cache skew short and cheap, so a 30% hit rate is not a 30% saving. Count it in GPU-seconds.
• Looking only at mean latency. Continuous batching improves the mean while widening the spread. Complaints come from p99.
• Not asking what "tokens per second" refers to. Per request, or summed across the server? The same setup differs by more than 10×, and most of the disagreement between benchmark posts lives here.
• Not counting retries and agent loops. Automatic retries on failure and multi-step tool calls can be a large share of the bill. Reducing $N$ starts here.
• Setting max context length to the maximum "just in case." It quietly eats your concurrency and drags $B$ down.

How to answer when you're asked. If someone says "halve our inference cost," don't lead with technique names. Decompose to $C = p N (T_p + L T_d) / B$, show from measurement which letter is bloated, and present the levers in order of least quality risk first. That is the answer with a spine.

Summary

• Cost decomposes into price × arrivals × work-per-request ÷ how well you batch, and there are only four kinds of moves
• Order by whether you're gambling with quality, not by size of win. Caching and batching pay without touching the output
• Quantization and distillation work, but they're irreversible and come with an obligation to re-evaluate. Hence last
• Some levers, like speculative decoding, only pay when the batch is empty. Rank depends on environment
• Add one lever at a time and measure against a frozen evaluation set. Do it all at once and you'll never find the cause