Prompt caching in production: why block order beat cutting text, and what changed with GPT-5.6

Every major model provider now discounts input tokens it has seen before. OpenAI's prompt caching guide says cached input can be "discounted up to 95%". Anthropic bills cache reads at a tenth of the base input price or less. Google passes on savings automatically on Gemini 2.5 and newer.

So teams switch caching on and expect the bill to fall. Often it barely moves, and nothing in the dashboard says why.

This post is about why. It's based on our own product, ConGreeto, a website assistant we build and run in production. We'll cover what we measured, what turned out to be wrong, the fix (which cut no prompt text at all), how we instrument it now, and how the provider rules changed in 2026 in ways that matter to anyone who did the same fix a year ago.

We've written about AI running costs in general in what an AI agent actually costs to run. This post goes deeper on caching alone.

How prefix caching works, in one paragraph

A model provider caches the beginning of your prompt. If the next request starts with exactly the same tokens, it reuses the work it already did on them and charges you less. The match is a prefix: it runs from the first token to the first token that differs, and stops there. Everything after the first difference is full price, even if it's identical to last time. That one rule explains almost every caching problem we've seen.

There's a second rule: below a minimum length, nothing is cached at all. When we did this work, OpenAI's documented minimum was 1,024 tokens.

What we thought the problem was

We had a working figure that each chatbot reply cost about 25,000 input tokens. The obvious plan was to cut: retrieve fewer knowledge chunks, shorten the persona, trim the history.

Before cutting anything, we measured. We took the token counts the provider actually reported for 171 replies (from OpenAI's usage object, not our estimate). The mean was 3,757 tokens per reply, median 3,048, 90th percentile 6,239, maximum 6,873. The 25,000 figure dated from an old internal persona that had long since been removed. Nobody had re-checked it.

Then we measured each block of the prompt. Knowledge chunks averaged about 104 tokens, not the ~400 we'd assumed, because chunks split on paragraph breaks and most were far shorter than the size cap. The whole retrieval section came to about 1,200 tokens. Cutting retrieval, the plan we nearly shipped, would have saved little and made answers worse.

Lesson one: measure per block before you cut anything. A total token count tells you how big the bill is. It doesn't tell you which part to change.

What the problem actually was: order

Our prompt was assembled from about a dozen blocks. Simplified, the order was:

The assistant's persona and company profile (stable for a given chatbot)
Campaign details, if any (stable for the campaign)
A rolling summary of the conversation so far (changes every turn)
Industry conversation guidance (never changes)
Lead-capture policy (never changes)
Contact-privacy rules (never changes)
Greeting rules (never changes)
Retrieved knowledge and listings (changes every turn)
Per-turn instructions and recent history (changes every turn)

Blocks 4 to 7, about 845 tokens, never change. But they sat below block 3, which changes on every turn. The cached prefix ended at block 3, so we paid full price for those 845 unchanged tokens on every reply.

The fix was to move blocks 4–7 above the summary. We didn't cut or reword a single word of prompt text, so answer quality couldn't change. Measured on the assembled prompts:

Before

After

Cacheable prefix

~1,161 tokens

~2,006 tokens

Full-price part of each reply

~2,562 tokens

~1,717 tokens

Total prompt

~3,723 tokens

The full-price part fell by about a third. The total didn't change at all.

These per-block numbers are estimates from the assembled prompt (characters ÷ 4). We checked the estimator against the provider's reported totals and it came within about 1%. Shares between blocks hold up even when an estimator is slightly off, because the error applies to every block equally.

The silent floor

The reorder fixed something less obvious too. Several of our chatbots had short personas, and their stable prefix came in under 1,024 tokens. For them, caching had never worked at all. There was no error and no warning. The only symptom was the bill. Moving 845 stable tokens up put every chatbot comfortably over the line.

If your system prompt is short, check this first. A cache hit rate of exactly 0% on a short prompt is usually the minimum length, not a bug in your code.

What it's worth depends on the model

The same reorder saves different amounts on different models, because the cached discount differs. Here's the arithmetic for input cost only, with prices from OpenAI's pricing page as of October 2026, assuming every turn hits the cache and ignoring write charges (more on those below):

Model

Input / cached input per 1M

Input cost per reply, before

After

Change

gpt-4o-mini

$0.15 / $0.075 (50% off)

$0.000471

$0.000408

−13%

gpt-6-luna

$0.10 / $0.01 (90% off)

$0.000268

$0.000192

−28%

On gpt-4o-mini, which we ran when we made the change, cached tokens were only half price, so the reorder took about 13% off input cost. On a model with a 90% cached discount, the identical change is worth more than twice as much. The bigger the cached discount, the more your block order matters. These are calculations from our measured token counts, not measured bills.

How we measure it now

The real fix wasn't the reorder. It was making the problem visible, so that the next person to add a prompt block can't silently undo it.

Every block now carries two things at assembly time: a label and a cacheable flag. The list of labelled blocks is the same list that gets sent to the model, so the accounting can't drift from the real request. Then, for every reply, we log one line:

[prompt] est_total=3723 cached_prefix=2006(54%) full_price=1717(46%) blocks=11 | persona=1150(31%) knowledge=520(14%) ...

(Illustrative numbers in the format we use.)

The important detail is how the prefix is counted. It isn't the sum of blocks marked cacheable. It's the unbroken run of cacheable blocks from the top, stopping at the first block that changes. Those two numbers differ in exactly one situation: when someone puts a changing block above a stable one. That's the regression we want to catch, so we log a warning when it happens:

prefix = 0
for block in blocks:
if not block.cacheable:
break
prefix += block.tokens

stranded = sum(b.tokens for b in blocks if b.cacheable) - prefix
if stranded:
log.warning("%d cacheable tokens sit below a per-turn block "
"and are billed at full price every turn", stranded)
if 0 < prefix < MIN_CACHEABLE_TOKENS:
log.warning("cacheable prefix %d is below the provider minimum; "
"nothing will be cached", prefix)

Two warnings, two silent failures turned into log lines. It costs one len() per block, needs no tokenizer and touches no network, so it stays on in production.

Alongside it we log the provider's own figures on every call: prompt tokens, cached tokens and hit rate. Our estimate says what should be cached. The provider says what was. If they disagree, something changed.

Lesson two: give every prompt block a label and a stability flag, and alert when a stable block lands below a changing one. Most caching regressions come from a new block added in the wrong place by someone who never knew the order mattered.

What changed in 2026

We made this fix in September 2026, on gpt-4o-mini, under OpenAI's caching rules at the time: automatic, no extra charge to write the cache, and the cached prefix could end anywhere. Since then OpenAI has retired the GPT-4o family (shutdown 14 January 2027) and changed how caching works on newer models. If you did a reorder like ours on an older model, check it again.

From OpenAI's prompt caching guide, checked 27 October 2026, for GPT-5.6 and later:

Cache writes now cost money. Writing a 30-minute cache entry costs 1.25× the normal input rate, and a 12-hour entry costs 2.5×. Reads cost 0.1× (0.05× on the GPT-6.1 Sol and Terra models). OpenAI's own example: one 30-minute write plus nine reads costs 2.15× a single uncached input, against 10× for ten uncached requests. Caching still pays from the second read. A prompt that's only ever sent once now costs more with caching than without.
The cache can only end at certain points. In implicit mode, a later request can match at the end of the latest message, up to 20 earlier message endings (user messages and the last of a run of tool results), the end of the first run of developer messages, and any explicit breakpoints you set. The guide is direct about the trap: a shared prefix that doesn't end at one of those boundaries isn't reusable on its own.
You can now place breakpoints yourself. In the Responses API, an explicit prompt_cache_breakpoint on a content block marks where a stable section ends. A request can create up to four cache writes, and the implicit breakpoint uses one of them. There's also an explicit-only mode, where nothing is cached unless you choose the breakpoint.
The minimum is 1,024 visible input tokens, not counting OpenAI's hidden system content. On earlier models, the guide now says the minimum varies with request settings such as tools, images and reasoning effort.

Read the second point against our prompt. Our stable blocks sit at the top, but the changing blocks below them are also system messages, sent in the same run. On the old rules, the cache ended wherever the content first changed, so the reorder was enough. On the new rules, the end of our stable section isn't a message boundary, so ordering alone may not get it cached. The likely fix is a boundary right after the stable section: an explicit breakpoint, or a separate message that ends there.

We're moving ConGreeto to a GPT-6 model now. As we write this, we haven't measured its cache rate in production on the new model, so we won't claim a result. The point for your system is the same either way: on newer models, order is necessary but not sufficient. The stable part has to end at a boundary the provider recognises.

The guide makes a few more points worth noting:

Summaries can break reuse. Appending new messages keeps the earlier prefix intact. Summarising, compacting or truncating earlier turns rewrites it. If you keep a rolling summary as we do, put it below everything stable.
Keep tool definitions identical. To limit which tools the model may use on a turn, use tool_choice or allowed_tools rather than removing definitions, which changes the prefix.
Busy prefixes overflow. Caches live on individual machines. The guide says traffic above about 15 requests per minute for one prefix and key can overflow to other machines, and suggests splitting high-volume traffic across several prompt_cache_key values.
Cached tokens still count toward rate limits. Caching lowers the bill, not your tokens-per-minute usage.

How the other providers handle it

The principle is the same everywhere: stable content first, at a boundary the provider recognises. The details differ. Checked 27 October 2026:

OpenAI (GPT-5.6+)

Anthropic Claude

Google Gemini

On by default?

Yes (implicit)

Opt in: one top-level cache_control, or cache_control on blocks

Implicit on 2.5 and newer

Where the cache can end

Message endings, first developer run, explicit breakpoints

At breakpoints you set (up to 4)

Prefix match

Minimum

1,024 tokens

512 tokens on current models; 1,024–4,096 on some older ones

2,048 on 2.5 models; 4,096 on newer ones

Write cost

1.25× (30 min) / 2.5× (12 h)

1.25× (5 min) / 2× (1 h)

Not stated on the caching page

Read cost

0.1× (0.05× on 6.1 Sol/Terra)

0.1× on most models; 0.05× or 0.025× on some newer ones

Discount applied automatically

Where to check

cached_tokens in usage

cache_read_input_tokens, cache_creation_input_tokens

usage.total_cached_tokens

Two Anthropic details catch teams out. First, its prompt caching docs build the cache in a fixed order: tools, then system, then messages. Changing a tool definition invalidates everything after it, and adding or removing an image invalidates the system and message caches. Second, when looking for an earlier cache entry, a breakpoint checks back at most 20 positions. In a long conversation with a single breakpoint at the end, an older cached prefix can fall out of reach. Put a breakpoint after the stable content, not just at the end.

On Google's newest models, the Gemini caching docs say a follow-up turn is cached in full only if it adds at least 2,050 tokens. Short follow-ups reprocess the last 2,050 tokens at the normal rate. Their advice matches everyone else's: put large, common content at the start, and send requests with a similar prefix close together.

Prompt caching checklist

Measure provider-reported tokens per request before deciding anything. Don't work from an estimate someone made months ago.
Break the prompt into labelled blocks and measure each one.
Sort blocks by how often they change: never, per customer, per conversation, per turn. Most stable first.
End the stable section at a boundary your provider recognises: a message ending or an explicit breakpoint.
Check the minimum. If your stable prefix is under it, nothing is cached.
Keep the stable section byte-for-byte identical: no timestamps, visitor names or request IDs in it, and stable tool definitions.
Put summaries, retrieval and per-turn instructions last.
Log your expected prefix and the provider's cached count on every call, and alert when they disagree.
Re-check when you change model. Minimums, write charges and boundary rules differ between model generations, even at the same provider.
Never trade answer quality for a cache hit. Our saving came from order, so nothing had to be cut. If a change does cut text, test it on real conversations first.

Frequently asked questions

Why is my prompt cache hit rate 0%?

Usually one of three things: the stable part of your prompt is shorter than the provider's minimum (1,024 tokens on current OpenAI models), something near the top changes on every request (a timestamp, a user name, a summary), or the stable section doesn't end at a boundary the provider can cache to. Check them in that order.

Does prompt caching cost extra?

On newer models, writing the cache does. OpenAI charges 1.25× the input rate to write a 30-minute entry on GPT-5.6 and later, and Anthropic charges 1.25× for a 5-minute entry. Reads cost a tenth of the input rate or less, so caching pays for itself from the second read. A prompt sent only once costs more with caching than without.

How much does prompt caching save in production?

It depends on how much of each request is a stable prefix and on the model's cached discount. Deriv reported in July 2026 that an 85.8% cache hit rate cut input-token costs by 77%. Our own reorder took about a third off the full-price part of each reply. In input cost, that's roughly 13% on gpt-4o-mini and 28% on gpt-6-luna, by calculation.

Should I put the conversation summary at the top of the prompt?

No. A rolling summary changes every turn. Anything placed below it can't be cached. Put it after all your stable instructions.

Is prompt caching the same as caching the model's answers?

No. Prompt caching reuses the provider's work on the input. The model still generates a fresh answer every time. OpenAI's guide notes that caching doesn't change output generation.

Sources

OpenAI, Prompt caching guide, checked 27 October 2026.
OpenAI, API pricing, checked October 2026.
Anthropic, Prompt caching, Claude Developer Platform docs, checked 27 October 2026.
Google, Context caching, Gemini API docs, checked 27 October 2026.
Osama Ghazal (Deriv), Prompt caching and production AI agent costs, 23 July 2026.
Brillnex Systems, internal measurements on ConGreeto: provider-reported token usage across 171 replies (August 2026), per-block prompt estimates before and after reordering (September 2026). Internal, not a benchmark.

Related reading

Brillnex Systems builds AI agents and LLM applications for companies that need them to work, and keep costing what they should, in production. If your AI bill is growing faster than your usage, talk to our engineering team or see our AI agent development services.

Prompt Caching in Production (2026): Why Block Order Beat Cutting Text, and What Changed With GPT-5.6