Prompt Caching and Cost Optimization

This lesson on Prompt Caching and Cost Optimization is hands-on and example-driven. Implement Anthropic's ephemeral prompt caching across large reference documents, agent tool definitions, and multi-turn conversations. You will configure cache breakpoints in the Python SDK to reduce input token costs by up to 90% and decrease API latency by up to 85% across repetitive or stateful workloads.

What You'll Be Able To Do

• Inject ephemeral cache control blocks into large static system prompts and reference texts.
• Calculate cost-benefit ratios between cache creation premiums and cache read discounts.
• Inspect API usage response objects to verify cache creation and read token volumes.
• Attach cache breakpoints to agent tool schemas to eliminate repetitive overhead.
• Implement rolling two-point cache markers in multi-turn conversation loops.
• Validate prompt token counts against model-specific minimum cache thresholds before deployment.

Detailed Concept Walkthrough

1. Prompt Caching Economics and Architecture

Prompt caching allows developers to store recurring prompt prefixes on Anthropic's servers between requests, dramatically slashing latency and marginal inference costs.

• Pricing Mechanics: Writing a prompt prefix to the cache incurs a 25% cost premium over standard base input token pricing. Subsequent requests reading from that cached prefix receive a 90% discount relative to baseline input pricing.
• Latency Optimization: The model avoids re-evaluating the full attention matrix over the cached token prefix, reducing response generation start time by up to 85% on large contexts.
• Prefix-Matching Requirement: The cached sequence must match the exact token prefix from the very beginning of the request. Any alteration in text or structure before the cache checkpoint forces a new cache write.

import anthropic

client = anthropic.Anthropic()

# Cost model calculation
base_input_cost_per_m = 3.00  # Claude 3.5 Sonnet baseline ($/MTok)
cache_write_cost_per_m = base_input_cost_per_m * 1.25  # $3.75 / MTok
cache_read_cost_per_m = base_input_cost_per_m * 0.10   # $0.30 / MTok

# Net savings realized on 2nd request onwards
print(f"Cache Read: ${cache_read_cost_per_m}/MTok vs Base: ${base_input_cost_per_m}/MTok")

Key Takeaway: Cache writes require a 25% upfront surcharge but yield a 90% input cost discount on subsequent calls matching that exact prefix.

2. Large-Context Document Caching

Large, unchanging reference materials like books, codebases, or policy manuals can be pinned to memory to service rapid follow-up queries.

• Syntax Placement: Ephemeral caching is declared by passing a list of content blocks in the system parameter or message body, appending "cache_control": {"type": "ephemeral"} to the static block.
• Execution Flow: The initial API call processes the full document and writes the KV cache, taking normal processing time (e.g., 25.4s for 187k tokens). Subsequent API calls referencing the exact same cached block resolve in a fraction of the time (e.g., 2.38s).
• Storage Model: The cached block remains available in memory for an ephemeral 5-minute sliding window that resets on every cache hit.

import anthropic

client = anthropic.Anthropic()
with open("pride_and_prejudice.txt", "r") as f:
    book_text = f.read()  # ~187,000 tokens

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1000,
    system=[
        {
            "type": "text",
            "text": book_text,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[{"role": "user", "content": "Analyze the relationship between Elizabeth and Darcy."}]
)

Key Takeaway: Marking heavy reference content with ephemeral cache blocks transforms long-document querying from seconds-long bottlenecks into near-instantaneous turns.

3. Cache Thresholds, TTL, and Validation

Anthropic enforces minimum token sizes and a strict 5-minute sliding Time-to-Live (TTL) that dictate whether prompt caching engages.

• Minimum Token Thresholds: Prompts must meet minimum size requirements to trigger caching: 1,024 tokens for Claude 3.5 Sonnet and Claude 3 Opus, or 2,048 tokens for Claude 3 Haiku. Sub-threshold prompts silently bypass caching.
• Sliding 5-Minute TTL: Every successful read from the cache refreshes the 5-minute lifetime window. If more than 5 minutes elapse between queries, the cache evicts and the next call incurs a full cache write charge.
• Response Metadata Inspection: API usage metrics include cache_creation_input_tokens (tokens written) and cache_read_input_tokens (tokens retrieved at 90% discount) to verify hits.

# Inspect token usage metrics from the API response
usage = response.usage
print(f"Cache Creation Tokens: {getattr(usage, 'cache_creation_input_tokens', 0)}")
print(f"Cache Read Tokens:     {getattr(usage, 'cache_read_input_tokens', 0)}")
print(f"Standard Input Tokens: {usage.input_tokens}")
print(f"Output Tokens:         {usage.output_tokens}")

Key Takeaway: Always check response usage metadata to ensure your prompts exceed model token minimums and avoid exceeding the 5-minute TTL window.

4. Tool Schemas and Multi-Turn Rolling Caching

Agentic workflows and conversations require dynamic cache placement to preserve tool definitions and conversation history without full recomputations.

• Tool Schema Caching: Tool definitions can include "cache_control": {"type": "ephemeral"} on the final tool schema, caching the entire tool suite prefix for subsequent agent steps.
• Rolling Breakpoint Strategy: In multi-turn chat, static breakpoints fail as context grows. Developers maintain two cache points: one on the second-to-last user turn to read the prior history cache, and one on the latest user turn to write the extended history forward.
• Prefix Preservation: To maintain cache consistency, new turns must only be appended to the end of the messages list without mutating earlier message contents or roles.

messages = [
    {"role": "user", "content": "First prompt"},
    {"role": "assistant", "content": "First response"},
    # Mark second-to-last user message to read existing cache
    {"role": "user", "content": [{"type": "text", "text": "Second prompt", "cache_control": {"type": "ephemeral"}}]},
    {"role": "assistant", "content": "Second response"},
    # Mark newest user message to write updated history
    {"role": "user", "content": [{"type": "text", "text": "Third prompt", "cache_control": {"type": "ephemeral"}}]}
]

Key Takeaway: Manage multi-turn conversations by leapfrogging cache breakpoints across the final two user turns to maintain read-write continuity.

Topics Covered in Prompt Caching and Cost Optimization

1. Prompt Caching Economics (0:00 - 1:52) — Explains the cost structure of paying a 25% write premium for a 90% discount on cache reads.
2. Large Document Caching (1:52 - 4:15) — Demonstrates caching a 187k-token book to cut follow-up query latency from 25.4s to 2.38s.
3. Constraints, Thresholds, and TTL (4:15 - 5:51) — Outlines the 5-minute sliding TTL, model token minimums, and usage response metadata verification.
4. Agent Tool Caching (5:51 - 6:59) — Shows how to attach cache control blocks directly to tool definitions for agent workflows.
5. Multi-Turn Conversation Caching (6:59 - 10:53) — Implements a rolling two-point caching loop across sequential user turns in a chat context.

LLMs & Generative AI for Practitioners Cheat Sheet

• cache_control: {"type": "ephemeral"} — Designates content block as a prefix cache breakpoint

  {"type": "text", "text": "Static context", "cache_control": {"type": "ephemeral"}}
  

• response.usage.cache_creation_input_tokens — Reports tokens written to cache at 125% price

  tokens_written = response.usage.cache_creation_input_tokens
  

• response.usage.cache_read_input_tokens — Reports tokens read from cache at 10% price

  tokens_saved = response.usage.cache_read_input_tokens
  

• tools=[..., {"name": "t", "cache_control": {"type": "ephemeral"}}] — Caches all preceding and current tool definitions

  tools = [{"name": "search", "description": "web search", "input_schema": {}, "cache_control": {"type": "ephemeral"}}]
  

• Rolling cache assignment — Places cache control on last two user turns

  messages[-1]["content"] = [{"type": "text", "text": q, "cache_control": {"type": "ephemeral"}}]
  

Comparison Table

Common Pitfalls

• Mistake: Sending prompts below 1,024 tokens expecting caching benefits. Avoid: Verify total prefix length exceeds model-specific minimum thresholds before setting cache control.
• Mistake: Allowing more than 5 minutes between sequential conversational queries. Avoid: Keep follow-up calls within the 5-minute sliding window or anticipate cache re-creation costs.
• Mistake: Modifying text prior to a cache control marker across requests. Avoid: Keep all preceding tokens completely static to preserve exact prefix matches.
• Mistake: Using a single static cache marker in a growing multi-turn dialogue. Avoid: Dynamically shift cache markers across the second-to-last and latest user turns.

FAQs

• What happens if my prompt is below the model's token threshold? The API silently bypasses prompt caching and processes all tokens at standard input pricing without throwing an error.
• How does the 5-minute Time-to-Live (TTL) work? It is a sliding window that resets to 5 minutes each time a request successfully reads from that cached prefix.
• Can I cache agent tool definitions alongside system prompts? Yes, adding cache_control to tool definitions caches the preceding system instructions and tool schemas together as a prefix.
• How do I know if my request hit the cache? Inspect response.usage.cache_read_input_tokens; non-zero values confirm tokens retrieved from cache at the 90% discount.