Cutting Inference Cost in Practice — What to Do First

Quantization and distillation both work, but in most shops there are two or three levers left that cost you nothing in quality. Break the bill into four numbers, then reorder the work: cache, batch, shorten, compress. What each lever actually buys, and what breaks when you take them out of order.

Before you swap the bulbs, turn off the empty rooms

When your electricity bill triples, the first move is not replacing every bulb in the house with an LED. First you find out which rooms are burning it, then you switch off the lights nobody is using. Buying new fixtures comes last. Do it in the other order and you end up spending money and effort while the bill barely moves.

Inference cost has exactly this shape. "Let's quantize" and "let's distill into a smaller model" are the eye-catching moves, but in most deployments there are two or three levers ahead of them that work better and cost nothing in quality. This article is not a tour of the techniques themselves. It is about what order to pull them in, and what each one costs you on the way out.

The bill is made of four numbers

Whether you own the GPUs or call somebody's API, the structure of the payment is the same: a unit price for compute, times the time you occupied it. A month of it looks like this.

$C$ is the monthly cost, $p$ the price per GPU-hour, $N$ the number of requests in the month, $T_p$ the time to read one prompt (the prefill), $L$ the average number of output tokens, $T_d$ the time to emit one token, and $B$ how many requests you manage to process together — the effective batch size.

Stripped of the symbols, it is a sentence which says: price × (how many arrive × how much work each one is) ÷ how many you clear at once. That is the whole thing.

What makes the formula useful is that it immediately tells you there are only four kinds of moves. Shrink $N$ (don't call), shrink $L$ (make it shorter), shrink $T_d$ (make each token lighter), grow $B$ (batch them). Every cost-cutting technique in circulation lands on one of those. Which also means that when someone brings you a proposal, you can ask "which letter does this move?" and judge it on the spot.

And the axis that sets the order is not how big the win is. It is whether you are gambling with quality. Growing $B$ does not change a single character of the output; most ways of shrinking $T_d$ change the output itself. When two moves buy the same thing, take the one that isn't a gamble. That is the skeleton of the whole plan.

Move 0: Find out where the money is going

You cannot skip the measurement. Three things to look at.

The mix of requests. Break traffic down by endpoint and by feature, and rank it by GPU-seconds, not request count. A feature that looks trivial by volume often turns out to be half the bill because its outputs are long.

The distribution of input and output lengths. Report the median and p95, not the mean. Averages hide the picture where a small number of enormous requests are sitting on the GPU for a long time. The $L$ in the formula is a mean, but what usually hurts you lives in the tail.

Duplicate calls. How often does the same input arrive twice? Are retries or agent loops calling the model two or three times for one user action? It is not unusual for this to be tens of percent of traffic.

Move 1: Don't call the model at all

The cheapest inference is the one you never run, and by definition it costs nothing in quality. There are three tiers of it.

Exact-match cache. Normalize the input, and if you've seen it, return the stored output. It's a few dozen lines of code, and it pays off well for anything used in a FAQ-ish way. One condition: set temperature to 0 so generation is deterministic. Bolt a cache onto a sampling setup and you permanently freeze the artifact where the same question gets different answers depending on who asked first.

Prefix cache. If a hundred requests begin with the same system prompt, the computation for that shared span can be reused. The mechanism for keeping past keys and values around is the one covered in Understanding the KV Cache from Scratch; this is that mechanism shared across requests. In vLLM it is one flag, --enable-prefix-caching. The output does not change by a single bit, and for setups with a long system prompt the prefill time simply disappears. If you change one setting today, change this one.

Semantic cache. Embed the input, and if a close-enough past question exists, return its answer. Hit rates go up — and this is the tier where you start gambling.

The failure mode is predictable. "Tomorrow's weather in Tokyo" and "tomorrow's weather in Osaka" sit very near each other in embedding space. So do "does this contract auto-renew?" and "does this contract not auto-renew?" Swap a proper noun, a number, or a negation and the meaning inverts while the distance barely moves. The workable compromise is a tight threshold plus a rule that excludes requests containing proper nouns or figures from the cache entirely. Your hit rate drops, but confidently returning the wrong answer costs far more than a miss.