Big Picture 12: Caching, Cost, Streaming, Errors, Structured Outputs and Batch

This document covers the API topics a production app needs: prompt caching and its worked cost example, streaming events, error and retry paths, structured outputs and the Batch API.

Prompt caching, request 1 vs request 2

The cache is a prefix match. However much of the request is identical byte by byte from the start, that much comes from the cache.

• Request 1 (first time): Tools + System prompt + Policy documents (50K tokens) are written to the cache at 1.25x. Question 1 is charged at the normal price.
• Cache breakpoint: everything before it is a cache write.
• Request 2 (within 5 min): Tools + System + Documents come from the cache, read at 0.1x (0.05x on Opus 5.5), plus question 2.
• Request 3: the timestamp in the system prompt changed. The tools still hit, but from the system onward everything misses and is written again.
• Order is fixed: tools, system, messages. Stable things on top, changing things below.
• Rules: at most 4 breakpoints, a minimum of about 512 to 4096 tokens depending on the model, and a TTL of 5 min (refreshed on read) or 1 hour.
• Silent cache-breakers: datetime.now() in the system prompt, changing the order of tools, unsorted JSON, a different tool set per request, switching models.
• How to check: is usage.cache_read_input_tokens greater than 0? If it is 0, there is a breaker somewhere. Bonus: cached tokens do not count toward ITPM.

A cost worked example

Claude Opus 5.5 prices per 1M tokens: input $4, output $20, cache write $5, cache read $0.20. Each request has 50,000 tokens of policy docs, a 500 token question and 1,000 tokens of output (thinking + text).

• Opus 5.5 prices (per 1M tokens): input $4, output $20, cache write $5, cache read $0.20.
• Each request: 50,000 tokens of policy docs + 500 tokens of question + 1,000 tokens of output (thinking + text).
• No cache (every request): 50K x $4 = $0.200, question $0.002, 1K x $20 = $0.020, so $0.222. Ten requests cost $2.22.
• Cache, first request: 50K x $5 = $0.250 (write) + $0.002 + $0.020 = $0.272.
• Cache, every later request: 50K x $0.20 = $0.010 (read) + $0.002 + $0.020 = $0.032.
• 10 requests total: $0.272 + 9 x $0.032 = $0.56, roughly 75% saved.
• Output is expensive, and Batch: an output token costs 5x an input token, and thinking counts as output too, so lower effort means a lower bill. The Batch API cuts 50% if you can wait up to 24 hours.
• Model and extras: Haiku 5.5 ($0.10 / $0.50) is about 40x cheaper, but check quality with evals. Web search is $10 per 1,000, images are about 1.3K tokens per MP, and US-only inference is 1.1x.

Streaming with SSE events

With stream: true the response arrives as server-sent events. The UI can show text immediately, and long requests do not time out.

• message_start: message ID, model name and initial usage.
• content_block_start: a new block begins: thinking, text or tool_use, with an index (0, 1, ...).
• content_block_delta: arrives again and again with the pieces: text_delta, thinking_delta and input_json_delta (pieces of tool input).
• content_block_stop: the block is complete, and the next block starts again.
• message_delta: the final stop_reason and output usage.
• message_stop: it is over.
• In between: ping events keep the connection alive, and an error event (for example overloaded) can show up mid-stream.
• Tool use in a stream: the tool's JSON input arrives in pieces (input_json_delta). Parse it only after joining all of it, then run the tool.
• In the SDK it is easy: Python stream.text_stream + get_final_message(), C# CreateStreaming (IAsyncEnumerable), TypeScript stream.on("text") + finalMessage().

When something goes wrong, errors and retries

A production app must handle every path. Either the HTTP request itself fails, or the request succeeds (HTTP 200) but the answer is incomplete or stopped.

• 400: the request is wrong (a bad param, format or unsupported model feature). Do not retry, fix the request.
• 401 / 403: wrong key or no permission. Check the key, workspace and scopes.
• 404 / 413: model not found or request too big. Check the model ID and the size (32 MB), or use the Files API.
• 429: rate limit (RPM / ITPM / OTPM). Wait as long as retry-after says and back off. A 429 without retry-after is the spend cap, so raise your tier.
• 500 / 529: server error or overloaded. Retry with exponential backoff.
• SDK retries: the SDK retries 2 times on its own (408, 409, 429, 5xx, network errors). The default timeout is 10 min. Catch from specific to general: RateLimit, then APIStatus, then Connection.
• stop_reason max_tokens: the answer was cut off. Raise max_tokens or send continue. If the tool input is incomplete, do not run the tool.
• stop_reason pause_turn: a long server-tool job. Send the response back as it is and the model continues.
• stop_reason refusal: a safety classifier stopped it. Check the category in stop_details. If fallbacks are on, another model is tried within the same call.
• Context window full: use compaction, context editing or shorten the old history.
• Tool error: send is_error: true with a message in the tool_result, and the model tries another way by itself.

Structured outputs, guaranteed valid JSON

Use structured outputs when a program has to parse the output (save to a DB, send to an API).

• 1 Give a schema: put a JSON schema in output_config.format, or use a Pydantic / Zod class in the SDK.
• 2 Constrained generation: while choosing each token, only tokens that keep the JSON valid against the schema are allowed. Invalid paths are closed.
• 3 Valid JSON: required fields and correct types, so parsing does not fail.
• 4 Typed object: messages.parse() gives parsed_output, a validated class.
• Example: schema {name, email, plan: "Free" | "Pro" | "Enterprise"}. The output always comes in this shape.
• Three approaches compared: just writing "give JSON" in the prompt is no guarantee. Structured outputs: yes. strict: true on a tool: yes (the tool input matches its schema).
• Caveat: the format is guaranteed, the truth is not, so do business validation. It does not work with citations in the same request, and prefill is disabled on new models.

Batch API, thousands of requests at half price

Use it when the answer is not needed right away: nightly reports, bulk classification, evals, data enrichment.

• 1 Create requests: each has a custom_id + normal Messages params. Up to 100,000 in one batch.
• 2 Submit: POST /v1/messages/batches, and you get a batch ID immediately.
• 3 Processing: processing_status is in_progress. Anthropic runs it on its own capacity, mostly quickly, 24 hours at most.
• 4 Poll / wait: batches.retrieve(id) until it has ended.
• 5 Results: each is succeeded, errored, canceled or expired.
• 6 Match by custom_id: results come back in any order, so never trust the position. Retry the failed ones.
• Benefit: both input and output are 50% cheaper, and it works with prompt caching too.
• When not to use: a user is waiting live (chat), or you need fast mode.
• Limits: batches have their own rate limits (how many requests can be queued). Each request is just like a normal Messages request.
• Example: sentiment for 10,000 reviews with Haiku + batch + structured output.