Case 4 — Optimize Token Costs for a High-Volume App

# Case 4 — Optimize Token Costs for a High-Volume App > *"Your RAG chatbot serves 500K requests/month at $30K/month total LLM cost. CFO wants you to halve it without quality regression. Walk through the optimization plan."* --- ## Step 1 — Measure where the cost is ```sql -- per-request cost breakdown SELECT AVG(input_tokens) AS avg_input, AVG(output_tokens) AS avg_output, AVG(input_tokens * 3.0 / 1e6) AS avg_input_cost, AVG(output_tokens * 15.0 / 1e6) AS avg_output_cost, AVG(input_tokens * 3.0 / 1e6 + output_tokens * 15.0 / 1e6) AS avg_total_cost FROM llm_requests WHERE served_at >= NOW() - INTERVAL '7 days'; ``` Result: - Avg input: 5500 tokens. - Avg output: 250 tokens. - Avg input cost: $0.0165. - Avg output cost: $0.00375. - Total: $0.020 per request. At 500K requests/month: $10K (input) + $1.9K (output) = ~$12K. Plus other overhead. Tracks with $30K total. **Input dominates.** Where in the input? ```sql SELECT AVG(system_tokens) AS sys, AVG(rag_context_tokens) AS rag, AVG(user_message_tokens) AS user_msg, AVG(tool_def_tokens) AS tools FROM llm_requests_detailed; ``` Result: - System: 500 tokens. - RAG context: 4000 tokens. - User message: 200 tokens. - Tool defs: 800 tokens. RAG context is 73% of input cost. Tools are 14%. System is 9%. ## Step 2 — Optimization plan Prioritized list: 1. **Prompt caching** — system + tool defs are static. Cache them. 2. **Tighter RAG retrieval** — currently 5 chunks × 800 tokens. Try 3 chunks × 600 tokens with reranking. 3. **Smaller model for triage** — use cheap model to detect simple cases. 4. **Cap output** — currently 250 avg; cap to 150 for the bulk of cases. Expected savings: - Caching: 9% + 14% = ~23% of input via cache (at 10% of price) → save ~$2K/month. - Tighter RAG: 5×800 = 4000 → 3×600 = 1800. Save 55% of RAG cost → ~$5K/month. - Smaller model for simple cases: 30% of requests fit; switch to GPT-4o-mini at 1/20th cost → ~$3K/month. - Output cap: small effect since output is already low-cost share. Combined: ~$10K/month saved. Hits the target. ## Step 3 — Implement prompt caching ```python response = claude.messages.create( model="claude-sonnet-4-6", system=[ { "type": "text", "text": SYSTEM_PROMPT + KNOWLEDGE_BASE + EXAMPLES, "cache_control": {"type": "ephemeral"} } ], tools=tools, # Anthropic caches tools automatically when system is cached messages=[ {"role": "user", "content": user_message + context} ], ) ``` Anthropic's prompt cache: ephemeral (5-minute TTL). For high-traffic apps, calls land within TTL → cache hit. For lower traffic, set up a keepalive job pinging the cache every 4 minutes. ## Step 4 — Tighter RAG Current: top 5 chunks × 800 tokens = 4000 tokens. New: retrieve top 20 candidates → rerank → top 3 chunks × 600 tokens = 1800 tokens. ```python def get_retrieval_context(query, top_k_initial=20, top_k_final=3, max_chunk_tokens=600): candidates = hybrid_search(query, top_k=top_k_initial) reranked = rerank(query, candidates, top_k=top_k_final) # Truncate each chunk if needed return [c[:max_chunk_tokens * 4] for c in reranked] # rough char→token ``` Validation: re-run eval set. Make sure quality didn't drop. ## Step 5 — Triage with a smaller model For simple FAQ-style questions, GPT-4o-mini is sufficient. Use it for triage: ```python def classify_complexity(user_message): response = gpt.chat.completions.create( model="gpt-4o-mini", messages=[{ "role": "user", "content": f'''Is this customer message a simple FAQ (one of: shipping status, refund request, account access) or complex (multi-part, ambiguous, frustrated tone)? Message: {user_message} Output ONLY 'simple' or 'complex'.''' }], max_tokens=10, ) return response.choices[0].message.content.strip().lower() def smart_route(user_message, ...): complexity = classify_complexity(user_message) if complexity == "simple": return gpt_4o_mini_handler(user_message, ...) # cheaper, fewer chunks, shorter responses else: return claude_sonnet_handler(user_message, ...) # default full RAG ``` Triage cost: ~$0.0001 per request. Negligible. Savings: 30% of traffic shifts to the cheaper model. ## Step 6 — Validate quality Before celebrating savings, validate quality didn't drop: ```python # Run eval on the new pipeline results_new = run_eval(new_pipeline, eval_set) results_old = run_eval(old_pipeline, eval_set) print(f"Old: accuracy {results_old.pass_rate}, cost per request ${results_old.avg_cost}") print(f"New: accuracy {results_new.pass_rate}, cost per request ${results_new.avg_cost}") ``` If quality dropped meaningfully (>2% accuracy), reverse the most-aggressive optimization. Target is "cost cut, quality maintained", not "cost cut at any quality cost". ## Step 7 — Monitor in production Set up dashboards: ```sql SELECT DATE(served_at) AS day, COUNT(*)