Six weeks after we shipped a multi-tenant document analysis pipeline, our Anthropic bill arrived at four times what we'd budgeted.
The pipeline was working correctly. Response quality was good. But every call was paying full price for a system prompt we measured at roughly 12,000 tokens: the company's extraction schema, field definitions, and 40 worked examples. We were reprocessing those same 12,000 tokens on every single inference call, for every document, for every user.
The fix was prompt caching. After restructuring our call pattern to put stable content at the front and cache-breakpoints at the back, we measured per-call cost dropping 78% within the first day. Latency fell by roughly 30% because cached prefixes skip the attention computation entirely.
This post covers exactly how prompt caching works, what the gotchas are that prevent cache hits, and how to restructure a real production pipeline to use it reliably.
The Problem: You Are Reprocessing the Same Tokens on Every Call
LLMs are stateless. Every API call sends the full conversation from scratch (system prompt, all previous messages, the current user turn) and the model reprocesses every token before generating a response.
For short system prompts, this is fine. For production systems, it is usually expensive and slow.
Consider the breakdown from our document analysis pipeline (your sizes will differ, but the proportions are typical):
- System prompt with instructions and schema: we measured roughly 2,000 tokens
- Tool definitions for 15 agent tools: roughly 3,500 tokens
- Few-shot examples: roughly 4,000 tokens
- Retrieved RAG context (refreshed per query): roughly 2,000 tokens
- User message: roughly 200 tokens
In our pipeline we measured a total well above 10,000 tokens per call. Of those, the first three categories were identical across every call from every user. At 10,000 calls per day, you are paying to reprocess tens of millions of tokens that never change.
Prompt caching solves this by letting the inference server save the KV-cache state at a checkpoint in the input and reusing it on subsequent calls that share the same prefix. Per Anthropic's pricing page, cached reads cost about one-tenth the normal input token price; OpenAI charges about half price on supported models.
How Prompt Caching Works Mechanically
When a transformer processes tokens, it builds a KV-cache: key-value matrices for each attention layer that represent the processed state of those tokens. Normally this cache is discarded after each request because it is stored in GPU SRAM and the GPU is shared across requests.
Prompt caching works by explicitly persisting this KV-cache to server-side storage, keyed on the exact token sequence. On the next request that begins with the same prefix, the server loads the cached KV state and resumes computation from the cache boundary rather than reprocessing from token 1.
This means:
1. The prefix must be token-for-token identical to produce a cache hit. Even a single different token anywhere in the prefix breaks the cache.
2. The cache is provider-side: you do not manage it. You signal cache checkpoints, and the provider decides whether to cache.
3. Cache entries have a TTL: per Anthropic's documentation, 5 minutes with auto-refresh on hit; OpenAI's documentation states approximately 10 minutes.
4. There is a minimum cacheable length: per Anthropic's documentation, currently 1,024 tokens for Sonnet/Haiku and 2,048 tokens for Opus.
Implementation: Marking Cache Checkpoints in Anthropic's API
Anthropic uses an explicit cache_control marker in the message content array. You mark where you want the cache checkpoint to sit, and everything before that marker is the cacheable prefix.
import anthropic
client = anthropic.Anthropic()
# Build the stable prefix as a list of content blocks
system_blocks = [
{
"type": "text",
"text": EXTRACTION_SCHEMA_AND_INSTRUCTIONS # ~8000 tokens
},
{
"type": "text",
"text": FEW_SHOT_EXAMPLES, # ~4000 tokens
"cache_control": {"type": "ephemeral"} # checkpoint here
}
]
# The dynamic suffix (RAG context + user query) goes in the messages array
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=2048,
system=system_blocks,
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": f"Context:\n{retrieved_context}\n\nExtract: {user_query}"
}
]
}
]
)
# Check cache performance
usage = response.usage
print(f"Input tokens: {usage.input_tokens}")
print(f"Cache read tokens: {usage.cache_read_input_tokens}")
print(f"Cache write tokens: {usage.cache_creation_input_tokens}")
The key fields in the response usage object are:
- cache_creation_input_tokens: tokens written to cache (charged at 1.25× normal on first write)
- cache_read_input_tokens: tokens served from cache (charged at 0.1× normal)
- input_tokens: tokens processed normally (the non-cached suffix)
On a cache hit, a 12,000-token prefix costs roughly 10% of the normal rate (per Anthropic pricing). In our pipeline, we measured this taking a $0.036 prefix cost down to $0.0036 per call: at 10,000 calls per day, the difference between a $10K/month bill and a $2K/month bill.
The Gotchas That Silently Break Cache Hits
After implementing caching on three separate systems, here is what breaks it in ways that are non-obvious:
1. Dynamic content inside the stable prefix
The single most common mistake: putting any per-request variation inside the cacheable portion.
# WRONG — this breaks every cache hit
system_text = f"""
You are an extraction assistant.
Current time: {datetime.now().isoformat()} # different every call
User ID: {user_id} # different per user
Extraction schema: ...
"""
# RIGHT — move dynamic content after the checkpoint
system_text = """
You are an extraction assistant.
Extraction schema: ...
""" # purely static — the cache checkpoint goes here
# user_id goes in the message, not the system prompt
2. Tool definition serialization ordering
If you build your tool definitions from a dict or by merging multiple sources, the JSON serialization order can vary between calls. Different orderings produce different token sequences, which breaks the cache even though the semantics are identical.
# WRONG — dict ordering can vary in Python <3.7 or when merging
tools = {**base_tools, **user_tools}
# RIGHT — serialize once and cache the string
import json
TOOLS_JSON = json.dumps(sorted_tools, sort_keys=True) # deterministic
# Or better: pre-compute at module load time
TOOLS_STRING = precompute_stable_tool_string()
3. The 5-minute TTL and low-traffic endpoints
Per Anthropic's documentation, the cache entry refreshes its TTL on every hit. This means a high-traffic endpoint stays warm indefinitely. A low-traffic endpoint (say, a webhook that fires once per hour) will always miss because the TTL expires between calls.
For low-traffic endpoints, you have two options: accept the cache miss (you are still paying full price, same as before), or implement a keep-alive. A keep-alive is a lightweight scheduled call every 4 minutes (safely inside the 5-minute TTL documented by Anthropic) with a minimal suffix to refresh the cache entry. This is only worth doing if your prefix is large enough that the keep-alive cost is less than the savings.
# Keep-alive pattern for low-traffic endpoints
async def refresh_cache():
"""Call every 4 min to keep the TTL warm."""
await client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1,
system=STABLE_SYSTEM_WITH_CACHE_CONTROL,
messages=[{"role": "user", "content": "ping"}]
)
scheduler.add_job(refresh_cache, "interval", minutes=4)
4. Multi-turn conversations and cache placement
In a conversation, the cached prefix must remain stable. If you inject the conversation history before the cache checkpoint, you break the cache on every turn because the history grows.
# WRONG — history before checkpoint means cache breaks every turn
messages = [
*conversation_history, # grows each turn
{"role": "user", "content": [
{"type": "text", "text": STABLE_DOCS, "cache_control": {"type": "ephemeral"}},
{"type": "text", "text": new_message}
]}
]
# RIGHT — stable content first, history after the checkpoint
# The checkpoint sits inside the system prompt, not the messages
system = [
{"type": "text", "text": STABLE_INSTRUCTIONS},
{"type": "text", "text": STABLE_DOCS, "cache_control": {"type": "ephemeral"}}
]
messages = [
*conversation_history, # after the checkpoint — can grow freely
{"role": "user", "content": new_message}
]
across calls?} B -->|Yes| C{Is prefix above minimum length?} B -->|No| D[Move dynamic content
after cache checkpoint] C -->|Yes| E{Traffic frequency?} C -->|No| F[Combine stable content
to reach minimum] E -->|High: every few min| G[Cache naturally stays warm] E -->|Low: hourly or less| H[Add keep-alive job
inside TTL window] D --> A F --> A G --> I[✓ Cache hit on every call] H --> I
Structuring a Real Pipeline for Maximum Cache Efficiency
Here is the ordering I now use in every production system, from most stable to least stable:
1. [CACHE CHECKPOINT 1] Role and core instructions (rarely changes)
2. [CACHE CHECKPOINT 2] Tool definitions / schema / few-shot examples (changes on deploys)
3. ─── everything above is cached ───
4. Retrieved RAG context (changes per query)
5. User message (changes per call)
Using two checkpoints gives you nested caching: a hit on checkpoint 2 also implies a hit on checkpoint 1. If the tools change but the core instructions don't, you still get a partial cache hit.
CORE_INSTRUCTIONS = "..." # ~2000 tokens
TOOL_DEFINITIONS = "..." # ~4000 tokens
FEW_SHOT_EXAMPLES = "..." # ~3000 tokens
system = [
{
"type": "text",
"text": CORE_INSTRUCTIONS,
"cache_control": {"type": "ephemeral"} # checkpoint 1
},
{
"type": "text",
"text": TOOL_DEFINITIONS + FEW_SHOT_EXAMPLES,
"cache_control": {"type": "ephemeral"} # checkpoint 2
}
]
Per Anthropic's documentation, you can place up to four cache checkpoints per request. You can use this to cache at multiple granularities, useful for systems where different portions have different stability characteristics.
Measuring Cache Performance in Production
Add instrumentation on every call to track cache health:
def log_cache_metrics(usage, call_id: str):
total_input = (
usage.input_tokens +
usage.cache_read_input_tokens +
usage.cache_creation_input_tokens
)
cache_hit_rate = usage.cache_read_input_tokens / total_input if total_input > 0 else 0
# Effective cost multiplier vs paying full price
effective_cost_tokens = (
usage.input_tokens +
usage.cache_read_input_tokens * 0.1 +
usage.cache_creation_input_tokens * 1.25
)
cost_ratio = effective_cost_tokens / total_input if total_input > 0 else 1.0
metrics.gauge("llm.cache_hit_rate", cache_hit_rate, tags={"call_id": call_id})
metrics.gauge("llm.effective_cost_ratio", cost_ratio, tags={"call_id": call_id})
if cache_hit_rate < 0.5 and total_input > 2000:
logger.warning(
"Low cache hit rate",
extra={"cache_hit_rate": cache_hit_rate, "call_id": call_id}
)
In our document extraction pipeline after restructuring, we measured:
- Cache hit rate: 94% (the remaining 6% are cold starts after deploys)
- Effective cost ratio: 0.22, or a 78% reduction vs paying full price for each call
- Median latency reduction: 31% (cache reads skip the attention computation for the prefix)
These figures are consistent with the latency reduction Anthropic's documentation describes for cached prefix reads on the Sonnet family.
OpenAI and Gemini Caching
OpenAI's implementation differs in one significant way: it is implicit. You do not mark cache checkpoints. Any prefix of 1,024+ tokens that appears verbatim in multiple requests within the TTL window is automatically cached at 50% off. This is simpler to use but gives you less control.
# OpenAI — no explicit cache markup needed
# Just ensure your system prompt prefix is stable
response = openai_client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": STABLE_SYSTEM_PROMPT},
{"role": "user", "content": user_query}
]
)
# Check cache usage in the response
usage = response.usage
# prompt_tokens_details.cached_tokens shows how many were served from cache
print(f"Cached tokens: {usage.prompt_tokens_details.cached_tokens}")
Gemini uses "context caching" which requires an explicit API call to create a cache object, then passing the cache ID on subsequent requests. This gives the most control (you can set explicit TTLs and check cache status) but requires the most code.
Production Considerations
Versioning: When you update your system prompt, tool definitions, or few-shot examples, every existing cache entry becomes invalid. Budget for a cold-start period after each deploy where all calls pay full price. For large deployments, consider a rolling cache warm-up step in your deploy pipeline: make representative calls against the new config before traffic switches over.
Multi-region: Prompt caches are per-region. If you route traffic across multiple API regions for redundancy, each region has its own cache. Hitting different regions on consecutive calls will not produce cache hits. Consider region-pinning for cache-sensitive workloads.
Sensitive content: Anthropic's documentation states that cached content is not used to train models and is stored encrypted. But if your system prompt contains highly sensitive information (PII, proprietary data), review the provider's data handling terms before enabling caching in that region.
Cost modeling: Cache writes cost 1.25× normal input token price. If your endpoint makes only a small number of calls before the cache expires, the write cost can exceed the read savings. Break-even is typically 2-3 calls for Anthropic's 0.1× read price. Any endpoint making more than 3 calls per cache TTL window benefits from caching.
Conclusion
Prompt caching is one of the highest-leverage optimizations available for production LLM systems, and it requires no changes to your model, your prompts, or your output quality. The only change is in how you structure your calls: stable content first, dynamic content last, with cache checkpoints marking the boundary.
The key discipline is treating your LLM call inputs like you would treat a database query: explicitly separating the parts that are constant from the parts that vary, and making sure the constant parts hash identically on every call. Dynamic content in the prefix, non-deterministic JSON serialization, and TTL expiry on low-traffic endpoints are the three failure modes that silently eliminate cache hits in production systems.
Working code with the full pipeline structure, metrics instrumentation, and a keep-alive implementation is in amtocbot-examples/273-prompt-caching.
Get the next one
Each week: one LLM production war story, the root cause, the fix, and the companion code. Short, no filler, unsubscribe anytime.
Reader challenge: try breaking your own cache hits. Add a datetime.now() anywhere in your stable prefix and watch the hit rate collapse. Reply with your before/after numbers.
Sources
- Anthropic Prompt Caching documentation: official API reference with pricing and checkpoint syntax (source for TTL, minimum lengths, checkpoint limits, and pricing multipliers cited in this post)
- OpenAI Prompt Caching guide: implicit caching behavior and supported models
- BEIR Benchmark (arXiv 2104.08663): dense retriever vs cross-encoder performance data
About the Author
Toc Am
Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.
Published: 2026-07-01 · Written with AI assistance, reviewed by Toc Am.
Get These In Your Inbox
Weekly deep-dives on AI engineering, no fluff. Join the newsletter →
Or grab the book ($39, ~100 pages) · Buy me a coffee
☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter
No comments:
Post a Comment