Recap — Working Within Token Budgets

# Working Within Token Budgets Every LLM call has a token cost: input tokens (your prompt + retrieved context + chat history) plus output tokens (model's response). Costs scale linearly; production apps live and die by their token discipline. ## The two sides of the budget ### Input tokens - System prompt (fixed overhead per call). - User message (variable). - Retrieved RAG context (often the dominant cost). - Tool definitions (function calling adds tokens). - Chat history (grows with conversation length). ### Output tokens - Whatever the model generates. - Usually 3-5x more expensive per token than input. For chat apps with long histories or RAG apps with big context retrieval, input dominates. For chatbots that generate long responses, output dominates. ## Context windows in 2026 | Model | Context window | |---|---| | Claude 4 (standard) | 200K tokens | | Claude 4 (1M context tier) | 1M tokens | | GPT-5 / GPT-4o | 128K-256K | | Gemini 2.5 Pro | 1M+ | | Llama 3 (depending on variant) | 8K-128K | | Most open small models | 8K-32K | 200K tokens ≈ 150K words ≈ a 600-page book. Plenty for most use cases. ## The "lost in the middle" problem Most LLMs pay disproportionate attention to: - The beginning of the context (system prompt, instructions). - The end of the context (most recent user message). Content in the middle of a long context window can be effectively ignored. Tested with extreme cases: put a needle (a specific fact) in the middle of 100K tokens of distracting content, ask the model to find it — performance drops noticeably. Implication for RAG: if you retrieve 20 documents and stuff them all in context, the middle ones might be ignored. Better to retrieve fewer, more relevant chunks. ## Token cost math (concrete examples) ### Example 1 — Simple chatbot - User message: 50 tokens. - System prompt: 200 tokens. - Model response: 300 tokens. - Total: 550 tokens. At GPT-4o ($3 input, $15 output per 1M): - Input: 250 × $3/1M = $0.00075. - Output: 300 × $15/1M = $0.0045. - Total: $0.0053 per request. At 10K requests/day: $53/day = ~$1600/month. ### Example 2 — RAG application - System prompt: 500 tokens. - Retrieved context (5 documents × 800 tokens each): 4000 tokens. - User question: 100 tokens. - Model response: 500 tokens. - Total: 5100 tokens. At Claude 4 Sonnet ($3 input, $15 output per 1M): - Input: 4600 × $3/1M = $0.0138. - Output: 500 × $15/1M = $0.0075. - Total: $0.0213 per request. At 10K requests/day: $213/day = ~$6400/month. ### Example 3 — Chat with growing history After 50 turns: ~50K tokens of history each call. At $3/$15 per 1M: - Input: 50K × $3/1M = $0.15 per call. - Output: 500 × $15/1M = $0.0075 per call. - Total: $0.16 per call. 100x cost of the simple chatbot. This is why "let conversation history grow unbounded" is a production bug. ## Strategies to control input tokens 1. **Truncate chat history.** Keep only last N turns or summarize older context. 2. **Cache common prefixes.** Anthropic's prompt caching (and equivalents at OpenAI, Gemini) lets you reuse a long system prompt at ~10% of original input cost. 3. **Tighter RAG retrieval.** Retrieve 3 chunks instead of 10; rerank for relevance. 4. **Compress system prompts.** Most prompts have flab — examples that aren't strictly needed, lengthy formatting instructions. Trim ruthlessly. 5. **Use cheaper models for triage.** Small model decides what to do; expensive model only runs when needed. ## Strategies to control output tokens 1. **Set max_tokens.** Hard cap. Stops runaway generation. 2. **Structured output.** Constraining to JSON/XML often produces shorter, focused outputs. 3. **Prompt for brevity.** "Reply in 2 sentences." "List 3 points, no preamble." 4. **Streaming.** Doesn't reduce cost but improves perceived latency. ## Prompt caching (the big 2026 win) All major providers now offer prompt caching. Cache a static prefix (system prompt, large knowledge base, examples) so subsequent calls reuse it cheaply. Anthropic example: cache a 50K-token document. First call: pay full price + small overhead to cache. Subsequent calls: ~10% the cost of the cached portion. For chatbots and RAG apps with repeated static context, this is a 5-10x cost reduction. Don't skip it. ## Production monitoring Track per-call: - input_tokens, output_tokens (most APIs return these). - Model version used. - Cache hit/miss. - Latency. In your warehouse: ```sql SELECT DATE(request_at) AS day, SUM(input_tokens * input_price + output_tokens * output_price) AS total_cost, AVG(input_tokens) AS avg_input, AVG(output_tokens) AS avg_output FROM llm_requests GROUP BY DATE(request_at); ``` When costs spike, drill into which feature changed input shape. ## Common token-budget mistakes 1. **Unbounded chat history.** Production cost balloons over weeks. 2. **No prompt caching for static prefixes.** Easy 5-10x cost reduction left on the table. 3. **Stuffing too much RAG context.** 20 documents in context: more cost, worse