Prompt Caching and Context Design — One Prefix Rule That Moves Your Bill by an Order of Magnitude

Are you paying to have the same system prompt re-read on every single request? Prompt caching only works on exact prefixes — and that one rule decides what goes where in your context. Why a single timestamp at the top wipes out everything below it, and how misreading the TTL can make caching 25% more expensive than not caching at all.

Reading the same manual out loud, every morning

Imagine a workplace where, every time you give a colleague a task, they first read a 300-page manual aloud from page one, cover to cover, and only then listen to what you actually wanted. Nobody would call that normal.

By default, that is exactly what a request to an LLM API does. The API is stateless. Your previous turns, your system prompt, your tool definitions, the internal document you want the model to consult — all of it is shipped again, in full, on every request. And the model reads all of it again from the top. "Reading," here, means pushing every token through attention to build its internal representation, and you are billed input-token rates for that compute.

Prompt caching removes the re-read. It lets you say: I already had you read this far last time — pick up from there. But the mechanism comes with exactly one very rigid constraint, and whether you understand it decides whether your bill for the same app differs by an order of magnitude.

Why it only works on prefixes

The constraint isn't an implementation quirk. It falls out of the Transformer itself.

Attention in a language model is causal: the token at position $t$ cannot see anything after itself. So its internal representation is determined entirely by the input from the start of the sequence up to $t$.

Equation (1) says one thing only: the K and V for token $t$ are built from tokens 1 through $t$ and nothing else. Whatever appears at position $t+1$ and beyond, those values do not change by a single bit.

The conclusion follows automatically. If two requests are byte-for-byte identical for their first $k$ tokens, the K and V for those $k$ positions are guaranteed to be identical too. There is nothing to recompute — you can simply reuse what you stored.

The reverse follows just as automatically. If there is even a one-token difference anywhere near the front, every $h_t$ from that position onward changes, so reuse stops one token earlier. This is prefix matching. Suffix matching and middle matching do not exist, and cannot. Swap out one document in the middle of your prompt and everything after it is recomputed from scratch.

Reusing K and V within a single generation is the subject of the KV cache article. Prompt caching is that same KV cache, kept alive across requests. Same principle, different lifetime.

The price list grows to three tiers

Turn caching on and your input tokens split into three rates. In Anthropic API terms the multipliers look like this (check the official pricing page for exact current numbers):

That last row — 0.1× — is where "order of magnitude" comes from. While you're hitting, that portion of the prompt costs a tenth. Which also means the ceiling on your savings is 10×, and no more.

The 1.25× write premium looks like it might eat into that, but the break-even arrives almost immediately. Use the same prefix twice and you pay $1.25 + 0.1 = 1.35$ against $2.0$ without caching. If a prefix gets used twice inside its lifetime, the write has already paid for itself.

Conversations grow quadratically

Caching really starts to matter once a conversation accumulates. It helps to count what is actually happening.

Take a 20-turn conversation: a 2,000-token system prompt, and each turn (user message plus model reply) adding 1,000 tokens of history. Turn $n$ therefore sends $2{,}000 + 1{,}000(n-1)$ input tokens. Summed over 20 turns:

$$\sum_{n=1}^{20}\bigl(2{,}000 + 1{,}000(n-1)\bigr) = 40{,}000 + 1{,}000 \times 190 = 230{,}000$$

In words: adding up every turn's input across the whole conversation comes to 230,000 tokens. The text sitting on screen when it's over is only 21,000 tokens — you bought it eleven times over. With $N$ turns the cumulative input scales as $N^2$, so the gap widens the longer people talk.