Cost and Performance Optimization

In production, both cost and latency matter. The Claude API has several levers for this — first the free wins (caching, batch), then the trade-offs (effort, a smaller model).

9.1 Prompt caching

If every request carries the same long prefix (system prompt, documents, tool definitions), cache it. The cache works on prefix match: the render order is tools → system → messages. If even one byte changes in the prefix, everything after it is invalidated.

Cache-friendly prompt structure

1.Tools (stable): same order, same definitions
2.System prompt (stable): no timestamp / random ID
3.Large documents (stable): ← cache breakpoint here
4.Conversation history: automatic caching covers the tail
5.New user question (volatile): changes every time — always at the end

Stable things on top, changing things below

r = client.messages.create(
    model="claude-opus-5-5", max_tokens=4000,
    cache_control={"type": "ephemeral"},          # automatic caching (recommended)
    system=[{"type": "text", "text": BIG_POLICY_DOC,
             "cache_control": {"type": "ephemeral"}}],   # explicit breakpoint
    messages=[{"role": "user", "content": question}],
)
print(r.usage.cache_creation_input_tokens, r.usage.cache_read_input_tokens)

Point	Value
Cache write cost	1.25x (5 min TTL) / 2x (1 hour TTL)
Cache read cost	~0.1x base input (0.05x on Opus 5.5) — 90%+ savings
TTL	Default 5 minutes, refreshed on every read; optional ttl: "1h"
Breakpoints	Maximum 4 per request
Min cacheable size	512–4096 tokens depending on the model
Verify	If usage.cache_read_input_tokens is 0, there is a silent invalidator (such as datetime.now() in the system prompt)

9.2 Other levers

Feature	What it does	Benefit
Message Batches API	Thousands of requests asynchronously, results within 24 hours	50% cheaper
Token counting	Count tokens beforehand with messages.count_tokens	Cost estimate, limit check
Effort tuning	low / medium on simple routes	Fewer tokens, lower latency
Model routing	Easy work to Haiku/Sonnet, hard work to Opus	Big cost cut
Fast mode	~2.5x faster output on Opus, premium price	Latency-critical apps
Compaction (beta)	The server summarizes the old part of a long conversation	The context window never fills up
Context editing (beta)	Clear old tool results / thinking	Save tokens in agent loops
Task budgets (beta)	Tell the agent a token budget and it paces itself	The agent finishes gracefully

batch = client.messages.batches.create(requests=[
    {"custom_id": f"review-{i}",
     "params": {"model": "claude-haiku-5-5", "max_tokens": 500,
                "messages": [{"role": "user", "content": f"Give the sentiment: {t}"}]}}
    for i, t in enumerate(reviews)])
# poll until processing_status is "ended", then:
for res in client.messages.batches.results(batch.id):
    print(res.custom_id, res.result.type)    # order is not guaranteed — match by custom_id