SUBMISSION: Case 3 — Token Cost Audit & Ephemeral Caching

Industry Context: A high-volume enterprise legal search assistant processes 500,000 queries per month against compliance policies and multi-turn user dialogues. The monthly LLM bill has ballooned to $30,000/month with median query latency at 4.2 seconds. The CFO has mandated a 50% cost reduction ($15,000/month cap) and a sub-2-second P95 latency target without degrading answer quality.

Deliverables

cost_analyzer.py: Telemetry script parsing request logs to break down input tokens, output tokens, system prompt overhead, and conversation history accumulation.
cached_rag_client.py: Client implementation integrating Anthropic ephemeral prompt caching (cache_control), sliding conversation window truncation, and prefix stabilization.
optimization_audit_report.md: Before-and-after audit report proving a >50% cost reduction and verifying zero regression on a 50-query golden benchmark.

Starter Code Scaffold

from typing import Dict, Any, List

def calculate_monthly_spend(requests: int, avg_input: int, avg_output: int,
                            input_price_per_m: float, output_price_per_m: float,
                            cache_read_ratio: float = 0.0,
                            cache_read_price_per_m: float = 0.30) -> Dict[str, float]:
    """Model monthly costs before and after prompt caching and context pruning."""
    base_input_cost = (requests * avg_input / 1_000_000) * input_price_per_m
    output_cost = (requests * avg_output / 1_000_000) * output_price_per_m
    # TODO: Calculate cached vs non-cached delta and net savings
    pass

100-Point Evaluation Rubric

Criterion 1: Telemetry & Cost Driver Decomposition (25 pts) — Granular breakdown isolating static system prompt, dynamic RAG chunks, and accumulating conversation history.
Criterion 2: Ephemeral Prompt Caching Architecture (25 pts) — Correct placement of cache_control blocks, static prefix isolation, handling 5-minute sliding TTL.
Criterion 3: Conversation Window & History Pruning (25 pts) — Sliding window truncation (last N turns) or summarizing older history to prevent quadratic cost explosion.
Criterion 4: Quality Preservation & SLA Verification (25 pts) — Verified latency reduction (<2.0s P95) and zero quality regression on the golden benchmark.

Implementation Guide & Case Steps

"Your RAG chatbot serves 500K requests/month at $30K/month total LLM cost. CFO wants you to halve it without quality regression. Walk through the optimization plan."

Step 1 — Measure where the cost is

-- per-request cost breakdown
SELECT
  AVG(input_tokens) AS avg_input,
  AVG(output_tokens) AS avg_output,
  AVG(input_tokens * 3.0 / 1e6) AS avg_input_cost,
  AVG(output_tokens * 15.0 / 1e6) AS avg_output_cost,
  AVG(input_tokens * 3.0 / 1e6 + output_tokens * 15.0 / 1e6) AS avg_total_cost
FROM llm_requests
WHERE served_at >= NOW() - INTERVAL '7 days';

Result:

Avg input: 5500 tokens.
Avg output: 250 tokens.
Avg input cost: $0.0165.
Avg output cost: $0.00375.
Total: $0.020 per request.

At 500K requests/month: $10K (input) + $1.9K (output) = ~$12K. Plus other overhead. Tracks with $30K total.

Input dominates. Where in the input?

SELECT
  AVG(system_tokens) AS sys,
  AVG(rag_context_tokens) AS rag,
  AVG(user_message_tokens) AS user_msg,
  AVG(tool_def_tokens) AS tools
FROM llm_requests_detailed;

Result:

System: 500 tokens.
RAG context: 4000 tokens.
User message: 200 tokens.
Tool defs: 800 tokens.

RAG context is 73% of input cost. Tools are 14%. System is 9%.

Step 2 — Optimization plan

Prioritized list:

Prompt caching — system + tool defs are static. Cache them.
Tighter RAG retrieval — currently 5 chunks × 800 tokens. Try 3 chunks × 600 tokens with reranking.
Smaller model for triage — use cheap model to detect simple cases.
Cap output — currently 250 avg; cap to 150 for the bulk of cases.

Expected savings:

Caching: 9% + 14% = ~23% of input via cache (at 10% of price) → save ~$2K/month.
Tighter RAG: 5×800 = 4000 → 3×600 = 1800. Save 55% of RAG cost → ~$5K/month.
Smaller model for simple cases: 30% of requests fit; switch to GPT-4o-mini at 1/20th cost → ~$3K/month.
Output cap: small effect since output is already low-cost share.

Combined: ~$10K/month saved. Hits the target.

Step 3 — Implement prompt caching

response = claude.messages.create(
    model="claude-sonnet-4-6",
    system=[
        {
            "type": "text",
            "text": SYSTEM_PROMPT + KNOWLEDGE_BASE + EXAMPLES,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    tools=tools,  # Anthropic caches tools automatically when system is cached
    messages=[
        {"role": "user", "content": user_message + context}
    ],
)

Anthropic's prompt cache: ephemeral (5-minute TTL). For high-traffic apps, calls land within TTL → cache hit. For lower traffic, set up a keepalive job pinging the cache every 4 minutes.

Step 4 — Tighter RAG

Current: top 5 chunks × 800 tokens = 4000 tokens.

New: retrieve top 20 candidates → rerank → top 3 chunks × 600 tokens = 1800 tokens.

def get_retrieval_context(query, top_k_initial=20, top_k_final=3, max_chunk_tokens=600):
    candidates = hybrid_search(query, top_k=top_k_initial)
    reranked = rerank(query, candidates, top_k=top_k_final)
    # Truncate each chunk if needed
    return [c[:max_chunk_tokens * 4] for c in reranked]  # rough char→token

Validation: re-run eval set. Make sure quality didn't drop.

Step 5 — Triage with a smaller model

For simple FAQ-style questions, GPT-4o-mini is sufficient. Use it for triage:

def classify_complexity(user_message):
    response = gpt.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": f'''Is this customer message a simple FAQ (one of: shipping status, refund request, account access) or complex (multi-part, ambiguous, frustrated tone)?

Message: {user_message}

Output ONLY 'simple' or 'complex'.'''
        }],
        max_tokens=10,
    )
    return response.choices[0].message.content.strip().lower()

def smart_route(user_message, ...):
    complexity = classify_complexity(user_message)
    if complexity == "simple":
        return gpt_4o_mini_handler(user_message, ...)  # cheaper, fewer chunks, shorter responses
    else:
        return claude_sonnet_handler(user_message, ...)  # default full RAG

Triage cost: ~$0.0001 per request. Negligible. Savings: 30% of traffic shifts to the cheaper model.

Step 6 — Validate quality

Before celebrating savings, validate quality didn't drop:

# Run eval on the new pipeline
results_new = run_eval(new_pipeline, eval_set)
results_old = run_eval(old_pipeline, eval_set)

print(f"Old: accuracy {results_old.pass_rate}, cost per request ${results_old.avg_cost}")
print(f"New: accuracy {results_new.pass_rate}, cost per request ${results_new.avg_cost}")

If quality dropped meaningfully (>2% accuracy), reverse the most-aggressive optimization. Target is "cost cut, quality maintained", not "cost cut at any quality cost".

Step 7 — Monitor in production

Set up dashboards:

SELECT
  DATE(served_at) AS day,
  COUNT(*) AS requests,
  AVG(input_tokens) AS avg_input,
  SUM(cost) AS daily_cost,
  AVG(cached_input_tokens::float / input_tokens) AS avg_cache_hit_rate,
  COUNT(*) FILTER (WHERE model_version = 'gpt-4o-mini') AS simple_routed,
  COUNT(*) FILTER (WHERE model_version = 'claude-sonnet-4-6') AS complex_routed
FROM llm_requests
GROUP BY day
ORDER BY day DESC LIMIT 30;

Track:

Daily cost (should drop ~50%).
Cache hit rate (should be > 85% for the static prefix).
Routing distribution (should match expected ~30% simple / 70% complex).
Quality proxies: user feedback ratings, complaint rate.

Step 8 — Report results

Result: monthly cost from $30K → $14K (53% reduction). Quality on eval set: 91% → 90% (within noise).

Key levers:

Prompt caching for static system+tools+examples: saved $2K/month.

RAG context optimization (20 → 3 with reranking): saved $5K/month.

GPT-4o-mini triage for simple cases: saved $3K/month.

Output token capping: saved $1K/month.

Other small optimizations: ~$5K/month.