SUBMISSION: Case 3 — Token Cost Audit & Ephemeral Caching

# SUBMISSION: Case 3 — Production Economics: Token Cost Audit, Ephemeral Caching & Latency Optimization for High-Volume RAG > **Industry Context:** A high-volume enterprise legal search assistant processes 500,000 queries per month against compliance policies and multi-turn user dialogues. The monthly LLM bill has ballooned to $30,000/month with median query latency at 4.2 seconds. The CFO has mandated a 50% cost reduction ($15,000/month cap) and a sub-2-second P95 latency target without degrading answer quality. --- ## Deliverables 1. `cost_analyzer.py`: Telemetry script parsing request logs to break down input tokens, output tokens, system prompt overhead, and conversation history accumulation. 2. `cached_rag_client.py`: Client implementation integrating Anthropic ephemeral prompt caching (`cache_control`), sliding conversation window truncation, and prefix stabilization. 3. `optimization_audit_report.md`: Before-and-after audit report proving a >50% cost reduction and verifying zero regression on a 50-query golden benchmark. --- ## Starter Code Scaffold ```python from typing import Dict, Any, List def calculate_monthly_spend(requests: int, avg_input: int, avg_output: int, input_price_per_m: float, output_price_per_m: float, cache_read_ratio: float = 0.0, cache_read_price_per_m: float = 0.30) -> Dict[str, float]: """Model monthly costs before and after prompt caching and context pruning.""" base_input_cost = (requests * avg_input / 1_000_000) * input_price_per_m output_cost = (requests * avg_output / 1_000_000) * output_price_per_m # TODO: Calculate cached vs non-cached delta and net savings pass ``` --- ## 100-Point Evaluation Rubric - **Criterion 1: Telemetry & Cost Driver Decomposition (25 pts)** — Granular breakdown isolating static system prompt, dynamic RAG chunks, and accumulating conversation history. - **Criterion 2: Ephemeral Prompt Caching Architecture (25 pts)** — Correct placement of `cache_control` blocks, static prefix isolation, handling 5-minute sliding TTL. - **Criterion 3: Conversation Window & History Pruning (25 pts)** — Sliding window truncation (last N turns) or summarizing older history to prevent quadratic cost explosion. - **Criterion 4: Quality Preservation & SLA Verification (25 pts)** — Verified latency reduction ( *"Your RAG chatbot serves 500K requests/month at $30K/month total LLM cost. CFO wants you to halve it without quality regression. Walk through the optimization plan."* --- ## Step 1 — Measure where the cost is ```sql -- per-request cost breakdown SELECT AVG(input_tokens) AS avg_input, AVG(output_tokens) AS avg_output, AVG(input_tokens * 3.0 / 1e6) AS avg_input_cost, AVG(output_tokens * 15.0 / 1e6) AS avg_output_cost, AVG(input_tokens * 3.0 / 1e6 + output_tokens * 15.0 / 1e6) AS avg_total_cost FROM llm_requests WHERE served_at >= NOW() - INTERVAL '7 days'; ``` Result: - Avg input: 5500 tokens. - Avg output: 250 tokens. - Avg input cost: $0.0165. - Avg output cost: $0.00375. - Total: $0.020 per request. At 500K requests/month: $10K (input) + $1.9K (output) = ~$12K. Plus other overhead. Tracks with $30K total. **Input dominates.** Where in the input? ```sql SELECT AVG(system_tokens) AS sys, AVG(rag_context_tokens) AS rag, AVG(user_message_tokens) AS user_msg, AVG(tool_def_tokens) AS tools FROM llm_requests_detailed; ``` Result: - System: 500 tokens. - RAG context: 4000 tokens. - User message: 200 tokens. - Tool defs: 800 tokens. RAG context is 73% of input cost. Tools are 14%. System is 9%. ## Step 2 — Optimization plan Prioritized list: 1. **Prompt caching** — system + tool defs are static. Cache them. 2. **Tighter RAG retrieval** — currently 5 chunks × 800 tokens. Try 3 chunks × 600 tokens with reranking. 3. **Smaller model for triage** — use cheap model to detect simple cases. 4. **Cap output** — currently 250 avg; cap to 150 for the bulk of cases. Expected savings: - Caching: 9% + 14% = ~23% of input via cache (at 10% of price) → save ~$2K/month. - Tighter RAG: 5×800 = 4000 → 3×600 = 1800. Save 55% of RAG cost → ~$5K/month. - Smaller model for simple cases: 30% of requests fit; switch to GPT-4o-mini at 1/20th cost → ~$3K/month. - Output cap: small effect since output is already low-cost share. Combined: ~$10K/month saved. Hits the target. ## Step 3 — Implement prompt caching ```python response = claude.messages.create( model="claude-sonnet-4-6", system=[ { "type": "text", "text": SYSTEM_PROMPT + KNOWLEDGE_BASE + EXAMPLES, "cache_control": {"type": "ephemeral"} } ], tools=tools, # Anthropic caches tools automatically when system is cached messages=[ {"role": "user", "content": user_message + context} ], ) ``` Anthropic's prompt ca