Prompt Caching and Context Design — One Prefix Rule That Moves Your Bill by an Order of Magnitude

Are you paying to have the same system prompt re-read on every single request? Prompt caching only works on exact prefixes — and that one rule decides what goes where in your context. Why a single timestamp at the top wipes out everything below it, and how misreading the TTL can make caching 25% more expensive than not caching at all.

Reading the same manual out loud, every morning

Imagine a workplace where, every time you give a colleague a task, they first read a 300-page manual aloud from page one, cover to cover, and only then listen to what you actually wanted. Nobody would call that normal.

By default, that is exactly what a request to an LLM API does. The API is stateless. Your previous turns, your system prompt, your tool definitions, the internal document you want the model to consult — all of it is shipped again, in full, on every request. And the model reads all of it again from the top. "Reading," here, means pushing every token through attention to build its internal representation, and you are billed input-token rates for that compute.

Prompt caching removes the re-read. It lets you say: I already had you read this far last time — pick up from there. But the mechanism comes with exactly one very rigid constraint, and whether you understand it decides whether your bill for the same app differs by an order of magnitude.

Why it only works on prefixes

The constraint isn't an implementation quirk. It falls out of the Transformer itself.

Attention in a language model is causal: the token at position $t$ cannot see anything after itself. So its internal representation is determined entirely by the input from the start of the sequence up to $t$.

Equation (1) says one thing only: the K and V for token $t$ are built from tokens 1 through $t$ and nothing else. Whatever appears at position $t+1$ and beyond, those values do not change by a single bit.

The conclusion follows automatically. If two requests are byte-for-byte identical for their first $k$ tokens, the K and V for those $k$ positions are guaranteed to be identical too. There is nothing to recompute — you can simply reuse what you stored.

The reverse follows just as automatically. If there is even a one-token difference anywhere near the front, every $h_t$ from that position onward changes, so reuse stops one token earlier. This is prefix matching. Suffix matching and middle matching do not exist, and cannot. Swap out one document in the middle of your prompt and everything after it is recomputed from scratch.

Reusing K and V within a single generation is the subject of the KV cache article. Prompt caching is that same KV cache, kept alive across requests. Same principle, different lifetime.

The price list grows to three tiers

Turn caching on and your input tokens split into three rates. In Anthropic API terms the multipliers look like this (check the official pricing page for exact current numbers):

That last row — 0.1× — is where "order of magnitude" comes from. While you're hitting, that portion of the prompt costs a tenth. Which also means the ceiling on your savings is 10×, and no more.

The 1.25× write premium looks like it might eat into that, but the break-even arrives almost immediately. Use the same prefix twice and you pay $1.25 + 0.1 = 1.35$ against $2.0$ without caching. If a prefix gets used twice inside its lifetime, the write has already paid for itself.

Conversations grow quadratically

Caching really starts to matter once a conversation accumulates. It helps to count what is actually happening.

Take a 20-turn conversation: a 2,000-token system prompt, and each turn (user message plus model reply) adding 1,000 tokens of history. Turn $n$ therefore sends $2{,}000 + 1{,}000(n-1)$ input tokens. Summed over 20 turns:

$$\sum_{n=1}^{20}\bigl(2{,}000 + 1{,}000(n-1)\bigr) = 40{,}000 + 1{,}000 \times 190 = 230{,}000$$

In words: adding up every turn's input across the whole conversation comes to 230,000 tokens. The text sitting on screen when it's over is only 21,000 tokens — you bought it eleven times over. With $N$ turns the cumulative input scales as $N^2$, so the gap widens the longer people talk.

Now run the same conversation with caching on. Each turn writes only what is newly appended (1.25×) and reads everything before it (0.1×). The writes total the conversation's own 21,000 tokens, once each: $21{,}000 \times 1.25 = 26{,}250$. The reads cover the remaining 209,000 tokens: $209{,}000 \times 0.1 = 20{,}900$. Together about 47,000, against 230,000 uncached — roughly one fifth.

The layout rule: static on top, volatile at the bottom

Everything above compresses into a single design rule. The less something changes, the closer to the front it goes.

In the Anthropic API a request is laid out as tools → system → messages. That outer order is fixed, so you design within it. Top to bottom:

1. Tool definitions (frozen until the next deploy)
2. System prompt, policy, tone and brand instructions
3. Few-shot examples
4. Stable reference documents (product manuals, terms, slices of a codebase)
5. Conversation history — append only
6. This turn's user message, the current time, any volatile metadata

And here is what reliably destroys the cache. Every one of these happens in real code:

• A timestamp at the top of the system prompt. The string changes on every request, so nothing below it matches. A one-token difference vaporizes tens of thousands of tokens
• A username or session ID at the top. The cache fragments into one copy per user, and none of them ever get warm
• RAG results injected above the system prompt. They change with every question, so the manual and the few-shot examples underneath are wasted too
• Re-summarizing the history every turn. The prefix is rewritten each time, so you pay full price each time
• Tool definitions generated from a dict. Unstable JSON key ordering is enough to break the match

If you genuinely need the time, put it in the trailing user message. Rounding it to a date is tempting, but the cache still dies the moment the date rolls over — so just move it to the bottom.

A breakpoint is a boundary, not a region

APIs with explicit control (Anthropic's cache_control, for instance) have you place a marker where you want caching. The common misreading is that only the block carrying the marker gets cached. It's the opposite: the marker is a boundary meaning everything from the start up to here. The number of markers is capped — four in the Anthropic API.

The standard move is to use two:

• One at the end of the static region (system prompt plus fixed documents). It is long-lived and shared by every request
• One at the end of the conversation history, a turn or two back from the newest message. It advances as the conversation does

That gives you a long-lived foundation and a growing history you can reuse separately. With only one marker you're forced to choose: rewrite the foundation every time the history grows, or give up on caching the history.

Some implementations do prefix matching automatically, with no markers at all, for prompts above a length threshold. The design discipline is identical there — you just don't have to declare anything.

Minimum lengths and block boundaries

Short prompts aren't cached at all. Each model has a documented minimum token count (typically in the 1,000–2,000 range). When someone reports that their carefully cached 500-token system prompt "does nothing," this is almost always why.

The other gotcha: most implementations match at block granularity, not token granularity. vLLM hashes in blocks of 16 tokens by default, so a trailing partial block never makes it into the cache. If behavior seems to flip over a difference of a few tokens, suspect this.

Don't guess whether it's working — read usage

You cannot tell by feel whether caching is engaged. It is reported in every response, so look there.

resp = client.messages.create(
    model=MODEL,
    system=[
        {"type": "text", "text": POLICY},                 # static
        {"type": "text", "text": MANUAL,
         "cache_control": {"type": "ephemeral"}},         # <- boundary of the cached span
    ],
    messages=history + [{"role": "user", "content": q}],  # volatile goes last
    max_tokens=1024,
)
u = resp.usage
print(u.cache_creation_input_tokens,  # written (large only on the first call)
      u.cache_read_input_tokens,      # hit (this is what should grow from call two on)
      u.input_tokens)                 # neither — plain input

If cache_read_input_tokens is still zero on the second request, your prefix changed by at least one token. Dump the fully assembled prompt from both calls and diff them from the front. Don't poke at the prompt hoping it fixes itself — find the exact position where the match broke. This is always mechanically findable.

Misread the TTL and caching costs you more

Caches expire. The default lifetime is a few minutes (five, in Anthropic's case, refreshed on every hit); you can opt into a longer TTL, but the write rate goes up. The only question that matters is how many times that prefix gets used within its lifetime — and getting it wrong makes your bill go up.

Take a 20,000-token fixed prefix hit once every 30 minutes, 48 times a day:

• No caching: $48 \times 20{,}000 \times 1 = 960{,}000$ units
• Default TTL (5 min): the interval exceeds the TTL, so every call misses and every call is billed as a write — $48 \times 20{,}000 \times 1.25 = 1{,}200{,}000$ units, 25% more expensive
• Long TTL (1 hour, 2×): hits keep refreshing it, so it survives — $20{,}000 \times 2 + 47 \times 20{,}000 \times 0.1 = 134{,}000$ units, roughly one seventh

Same workload; the TTL choice alone swings it between one seventh and 1.25×. The working rule is: on any path whose access interval is longer than the TTL, do not enable caching. A nightly batch that runs once a day is the textbook case.

If you host the model yourself

The same mechanism is available off-API. vLLM's automatic prefix caching and SGLang's RadixAttention both share KV blocks across requests. With no per-token billing, the payoff shows up as time rather than money: you skip the prefill entirely, so TTFT (time to first token) drops.

The conditions are identical: the match must be a prefix, it must land on block boundaries, and there has to be free GPU memory. When free blocks run out, older cache entries get evicted — so pushing concurrency too high quietly drops your hit rate. Watch hit rate alongside throughput during load tests.

How this shows up on the job

Who touches it, and when. The decisive moment belongs to the backend engineer assembling the context for an LLM application. This is front-of-design work: fixing it later ripples through prompt management, RAG, and history storage all at once. After that, it's whoever runs the inference platform chasing a cost anomaly — "why did this jump last month?" turning out to be one line added to the system prompt that split the cache is a classic.

Names you'll actually type. On the API side: cache_control, ephemeral, the TTL setting, and the response fields cache_creation_input_tokens and cache_read_input_tokens. Self-hosted: vLLM's --enable-prefix-caching, --block-size, gpu_memory_utilization, and the prefix cache hit rate metric. At minimum, put cache hit rate and per-request input token breakdown (plain / write / read) on the dashboard.

Failure modes worth knowing before they bite.

1. Timestamps, IDs, or anything random at the front. The cache dies while the app keeps working perfectly, so nobody notices until the invoice arrives
2. Caching on a path whose interval exceeds the TTL. As above: 25% more expensive than doing nothing
3. Per-user prefixes. Just hoisting the shared portion above the per-user portion makes hit rate jump
4. Summarizing history every turn. If you must compress, do it once at a natural break, then go back to append-only
5. Cost projections that assume a 100% hit rate. Redo the estimate with the measured rate, and compare it against the other levers in Cutting Inference Cost in Practice before choosing an order of attack
6. Assuming caching changes the output. It doesn't. Caching reuses computation; model behavior is identical. Confusing the two burns days on a quality regression that isn't there

The review question. "If you swap one document in the middle of the prompt, how many tokens get recomputed?" Answer: everything from that position to the end. Being able to answer that in one breath is the same as being able to explain why volatile content belongs at the bottom.

Takeaways

• Prefix-only matching isn't a limitation of the implementation — under causal attention, $h_t$ depends solely on tokens 1 through $t$, so it's a consequence of the architecture
• Reads cost 0.1×, so 10× is the ceiling on savings. Two uses within the lifetime already repay the 1.25× write
• Stack the context in order of stability. One volatile value at the top invalidates everything beneath it
• If the TTL and the access interval don't line up, enabling caching makes things worse. Choosing not to cache is part of the design

Context design isn't about making the model smarter — it's about what you pay for the same answer. The longer the context an application assumes, the more this matters, so it pairs naturally with the techniques behind long-context LLMs when you're deciding where to invest.