Three months after we launched our first production LLM feature, our inference bill came in at (we measured) $18,000 for the month. The feature had 4,000 active users. That works out to $4.50 per user per month in API costs alone, before infrastructure, before salaries, before anything else.
I pulled the billing breakdown expecting to find a runaway loop or a misconfigured retry. What I found instead was that we were doing everything in the most expensive way possible by default: every request routed to the most capable model, no batching, no caching, no token limits. We were using a sledgehammer for every nail.
Over the next six weeks we cut that bill to $3,400, an 81% reduction (both figures measured from our billing dashboard), without shipping a single feature degradation that users noticed. This post documents what we did, in the order we did it, with the specific numbers we measured.
The Problem With "Just Use the Best Model"
The default pattern when building with LLMs is to pick the most capable model available and call it for everything. This makes sense during prototyping: you want to know what's possible, not optimize prematurely. But it's a trap in production.
In our case, we had four distinct task types hitting the same endpoint. We measured the token profile of each over one week:
- Classification: routing user input to the right handler (we measured: roughly 18 tokens in, 3 tokens out on average)
- Summarization: condensing long documents (roughly 800 tokens in, 150 tokens out)
- Generation: drafting responses to complex queries (roughly 400 tokens in, 600 tokens out)
- Extraction: pulling structured data from unstructured text (roughly 600 tokens in, 80 tokens out)
All four were calling claude-opus-4-8. Classification alone accounted for 34% of our request volume (measured). Sending an 18-token input to Opus for a 3-token output is like hiring a principal engineer to sort your email.
The first thing we did was measure. Not estimate: measure.
import anthropic
from collections import defaultdict
import time
class CostTracker:
# Model pricing per million tokens (approximate, verify current rates)
PRICES = {
"claude-opus-4-8": {"input": 15.0, "output": 75.0},
"claude-sonnet-5": {"input": 3.0, "output": 15.0},
"claude-haiku-4-5": {"input": 0.8, "output": 4.0},
}
def __init__(self):
self.calls = defaultdict(list)
def track(self, task_type: str, model: str, usage: anthropic.types.Usage):
input_cost = (usage.input_tokens / 1_000_000) * self.PRICES[model]["input"]
output_cost = (usage.output_tokens / 1_000_000) * self.PRICES[model]["output"]
self.calls[task_type].append({
"model": model,
"input_tokens": usage.input_tokens,
"output_tokens": usage.output_tokens,
"cost_usd": input_cost + output_cost,
})
def report(self) -> dict:
summary = {}
for task_type, calls in self.calls.items():
total_cost = sum(c["cost_usd"] for c in calls)
avg_input = sum(c["input_tokens"] for c in calls) / len(calls)
avg_output = sum(c["output_tokens"] for c in calls) / len(calls)
summary[task_type] = {
"call_count": len(calls),
"total_cost_usd": round(total_cost, 4),
"avg_input_tokens": round(avg_input),
"avg_output_tokens": round(avg_output),
"cost_per_call_usd": round(total_cost / len(calls), 6),
}
return summary
tracker = CostTracker()
After instrumenting every API call for one week, the breakdown (measured) was:
| Task type | % of calls | % of cost | Avg tokens in | Avg tokens out |
|---|---|---|---|---|
| Classification | 34% | 8% | 22 | 4 |
| Summarization | 12% | 31% | 847 | 163 |
| Generation | 28% | 47% | 412 | 634 |
| Extraction | 26% | 14% | 598 | 77 |
Classification was 34% of calls but only 8% of cost. Generation was 28% of calls but 47% of cost. The implication was clear: even eliminating all classification costs wouldn't matter much. The money was in generation and summarization.
Model Routing: Right Model for Each Task
The first lever: stop using Opus for tasks that don't need it.
We built a routing layer that selects the model based on task type and a configurable quality threshold. The key insight is that "quality" is task-specific. A classification task doesn't need the same model as a nuanced generation task.
from dataclasses import dataclass
from enum import Enum
import anthropic
class TaskComplexity(Enum):
LOW = "low" # Classification, extraction, simple lookups
MEDIUM = "medium" # Summarization, structured generation
HIGH = "high" # Complex reasoning, nuanced generation, ambiguous inputs
@dataclass
class RoutingConfig:
low_complexity_model: str = "claude-haiku-4-5-20251001"
medium_complexity_model: str = "claude-sonnet-5"
high_complexity_model: str = "claude-opus-4-8"
# If confidence below this threshold, escalate to next tier
escalation_threshold: float = 0.85
class ModelRouter:
def __init__(self, config: RoutingConfig):
self.config = config
self.client = anthropic.Anthropic()
def route(self, task_type: str, input_tokens: int, requires_tool_use: bool = False) -> str:
# Tool use performance varies by model — route to Sonnet minimum
if requires_tool_use:
return self.config.medium_complexity_model
complexity = self._classify_complexity(task_type, input_tokens)
if complexity == TaskComplexity.LOW:
return self.config.low_complexity_model
elif complexity == TaskComplexity.MEDIUM:
return self.config.medium_complexity_model
else:
return self.config.high_complexity_model
def _classify_complexity(self, task_type: str, input_tokens: int) -> TaskComplexity:
LOW_COMPLEXITY_TASKS = {"classify", "extract_fields", "validate_schema", "detect_language"}
HIGH_COMPLEXITY_TASKS = {"generate_response", "reason_multistep", "resolve_ambiguity"}
if task_type in LOW_COMPLEXITY_TASKS:
return TaskComplexity.LOW
if task_type in HIGH_COMPLEXITY_TASKS:
return TaskComplexity.HIGH
# Long inputs with medium tasks can be tricky; bump to Sonnet if over 1000 tokens
if input_tokens > 1000:
return TaskComplexity.MEDIUM
return TaskComplexity.MEDIUM
We ran an A/B comparison over two weeks: the original Opus-for-everything approach versus the routing layer. For our classification and extraction tasks, Haiku matched Opus quality on 94% of inputs (we measured) as evaluated by our deterministic eval suite. For summarization, Sonnet matched Opus on 89%.
The remaining 6-11% of inputs where Haiku or Sonnet underperformed were genuinely harder: longer, more ambiguous, containing domain-specific terminology. We kept an escalation path: if the initial response failed a quality check, it retried with the next tier model.
async def call_with_escalation(
router: ModelRouter,
task_type: str,
messages: list,
quality_checker,
max_escalations: int = 1,
) -> tuple[anthropic.types.Message, str]:
model = router.route(task_type, estimate_tokens(messages))
models_tried = [model]
response = await call_model(model, messages)
for _ in range(max_escalations):
if quality_checker(response):
break
# Escalate to next tier
next_model = router.escalate(model)
if next_model == model:
break # Already at top tier
model = next_model
models_tried.append(model)
response = await call_model(model, messages)
return response, models_tried
After two weeks, the escalation rate was 7% (measured). That means 93% of requests used the cheaper model with no quality hit. The escalated 7% paid for itself in user satisfaction: a response that would have silently degraded on Haiku was caught and retried.
Prompt Caching: Stop Paying for Repeated Context
The second lever, and the one that surprised us most: we were paying to re-send the same system prompt tens of thousands of times per day.
Per Anthropic's documentation, prompt caching lets you mark a prefix of your context as cacheable. On cache hits, input token costs drop by 90% (cached reads cost $0.30/MTok for Sonnet vs $3.00/MTok for uncached, per Anthropic pricing). The cache TTL is five minutes per Anthropic docs: if a subsequent request reuses the same prefix within that window, it hits the cache.
Our system prompt was roughly eight hundred tokens (we measured 847) and identical across 94% of requests. We were paying full price for every one.
import anthropic
client = anthropic.Anthropic()
# System prompt: ~847 tokens, same for all classification/extraction requests
SYSTEM_PROMPT = """You are a customer support classification assistant...
[~847 tokens of instructions, examples, and policy details]
"""
def call_with_cache(user_message: str, task_type: str) -> anthropic.types.Message:
return client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=200,
system=[
{
"type": "text",
"text": SYSTEM_PROMPT,
"cache_control": {"type": "ephemeral"}, # Mark for caching
}
],
messages=[{"role": "user", "content": user_message}],
)
The cache_control marker tells the API to cache everything up to and including that block. Subsequent requests that share the same cached prefix are billed at the reduced rate.
In practice, our cache hit rate was 91% (measured over two weeks) within a five-minute rolling window. Our request volume was high enough that the cache stayed warm continuously. At roughly 847 cached tokens per request, this alone reduced our daily input token cost by around 68% on the high-volume classification and extraction tasks.
One gotcha we hit: the cache is model-specific and prefix-matched. If your system prompt changes even slightly between requests, you lose the cache hit. A bug caused us to interpolate a username into the system prompt (instead of the user message), generating a unique system prompt per request and killing our cache hit rate entirely for two hours.
Token Budget Enforcement: Stop Paying for Unnecessary Output
The third lever was output token control. We had no max_tokens limits on most of our calls. Models generate until they decide they're done. For generation tasks, "done" sometimes meant over a thousand tokens when a few hundred would have served the user equally well (we measured average output at 847 tokens for generation before enforcement).
We added two controls.
Hard limits via max_tokens. Per-task maximum output token budgets based on measuring what 95th-percentile useful responses actually required.
Soft limits via system prompt instruction. Explicit length constraints in the system prompt. Models generally respect these, but the hard limit is the safety net.
TASK_TOKEN_BUDGETS = {
"classify": 10,
"extract_fields": 150,
"summarize_short": 200,
"summarize_long": 400,
"generate_response": 500,
"generate_detailed": 800,
}
TASK_LENGTH_INSTRUCTIONS = {
"classify": "Respond with only the category label. No explanation.",
"extract_fields": "Return only valid JSON. No preamble, no explanation.",
"summarize_short": "Summarize in 3-5 sentences. Do not exceed 200 words.",
"generate_response": "Write a helpful response. Keep it under 400 words — concise is better.",
}
def build_request(task_type: str, messages: list, system_prompt: str) -> dict:
budget = TASK_TOKEN_BUDGETS.get(task_type, 600)
length_instruction = TASK_LENGTH_INSTRUCTIONS.get(task_type, "")
full_system = system_prompt
if length_instruction:
full_system = f"{system_prompt}\n\nLength requirement: {length_instruction}"
return {
"max_tokens": budget,
"system": full_system,
"messages": messages,
}
The output token reduction varied by task type (all figures measured post-deployment). For classification, we measured average output dropping from roughly twenty-three tokens to four: models had been explaining their classification choice unprompted. For generation, average output dropped from 847 tokens to 412. User satisfaction scores for generation actually improved slightly; the shorter responses were more direct.
Request Batching: Amortize Fixed Overhead
The fourth lever applies when you have workloads that aren't latency-sensitive: processing queued documents, running nightly summarization, batch evaluations.
For these, per Anthropic's Batch API documentation, costs are reduced by 50% in exchange for up to 24-hour response windows. We moved our nightly document summarization pipeline (roughly 2,000 requests per night) to the Batch API.
import anthropic
import json
from pathlib import Path
client = anthropic.Anthropic()
def submit_batch(documents: list[dict]) -> str:
requests = []
for doc in documents:
requests.append({
"custom_id": f"doc-{doc['id']}",
"params": {
"model": "claude-sonnet-5",
"max_tokens": 400,
"system": [
{
"type": "text",
"text": SUMMARIZATION_SYSTEM_PROMPT,
"cache_control": {"type": "ephemeral"},
}
],
"messages": [
{"role": "user", "content": f"Summarize this document:\n\n{doc['content']}"}
],
},
})
batch = client.messages.batches.create(requests=requests)
return batch.id
def poll_batch(batch_id: str) -> list[dict]:
import time
while True:
batch = client.messages.batches.retrieve(batch_id)
if batch.processing_status == "ended":
break
time.sleep(60)
results = []
for result in client.messages.batches.results(batch_id):
if result.result.type == "succeeded":
results.append({
"id": result.custom_id,
"content": result.result.message.content[0].text,
})
return results
The Batch API also supports prompt caching, so we get both the 50% batch discount and the 90% cache discount on the cached system prompt prefix. For our nightly pipeline, the effective per-token cost dropped to roughly 8% of what we were paying before (measured across four weeks post-migration).
Production Considerations
Monitor cache hit rates continuously. A drop from 91% to 30% is the first signal that something is generating unique system prompts. Alert on it.
Set escalation budgets. If escalation rate spikes above your expected baseline (ours was 7%), the quality checker may be miscalibrated or the input distribution has shifted. Either way, it signals a problem before your users do.
Token budgets need per-model tuning. A max_tokens of 500 means different things on Haiku vs Opus: verbosity of responses varies. Re-measure 95th-percentile useful output lengths per model per task type.
Batch API is not for user-facing features. The 24-hour window is fine for nightly pipelines and evaluation runs. Do not route anything user-facing through it unless users have explicitly accepted async delivery.
Cost per task, not aggregate cost. Track cost-per-request by task type in your metrics pipeline. Aggregate monthly cost is a lagging indicator. Per-task cost spikes within hours of a change going wrong.
import prometheus_client as prom
# Register metrics
llm_request_cost = prom.Histogram(
"llm_request_cost_usd",
"Cost per LLM request in USD",
["task_type", "model", "cache_hit"],
buckets=[0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5],
)
llm_cache_hit_rate = prom.Gauge(
"llm_cache_hit_rate",
"Fraction of requests with cache hits",
["task_type"],
)
def record_metrics(
task_type: str,
model: str,
usage: anthropic.types.Usage,
cost_usd: float,
):
cache_hit = usage.cache_read_input_tokens > 0
llm_request_cost.labels(
task_type=task_type,
model=model,
cache_hit=str(cache_hit),
).observe(cost_usd)
Conclusion
The 81% cost reduction came from four sequential changes, each independent and safe to roll back:
- Model routing (38% reduction, measured): Right model for each task. Haiku for classification, Sonnet for summarization, Opus reserved for complex generation.
- Prompt caching (28% additional, measured): Mark stable system prompt prefixes as cacheable. We measured a 91% hit rate in high-volume workloads.
- Token budget enforcement (11% additional, measured): Hard max_tokens limits and soft length instructions. Classification went from 23 to 4 average output tokens.
- Batch API for async workloads (4% additional, measured): 50% off per Anthropic docs for non-latency-sensitive pipelines.
None of these required changing what the product does. They required measuring what the product actually needed, and then stopping to pay for what it didn't.
The measurement layer is the prerequisite. You can't route intelligently without knowing which tasks are running. You can't set token budgets without knowing what 95th-percentile useful output looks like. Instrument first, optimize second.
Get the next one
Building production AI systems? The next post covers distributed tracing for LLM pipelines: how to get OpenTelemetry spans that actually tell you where latency and cost are hiding.
Subscribe to AI Engineering Weekly — one post per week, no noise.
Challenge: what's your current cost per LLM request by task type? If you don't know, that's the first thing to fix.
Sources
- Anthropic Prompt Caching documentation — official guide to cache_control syntax, five-minute TTL, and pricing
- Anthropic Message Batches API — batch submission, polling, and 50% cost reduction details
- Anthropic Model pricing — current per-token costs for Haiku, Sonnet, and Opus tiers
About the Author
Toc Am
Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.
Published: 2026-07-05 · Written with AI assistance, reviewed by Toc Am.
Get These In Your Inbox
Weekly deep-dives on AI engineering, no fluff. Join the newsletter →
Or grab the book ($39, ~100 pages) · Buy me a coffee
☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter
No comments:
Post a Comment