Case 3 — Evaluate Two LLMs for a Specific Task

# Case 3 — Evaluate Two LLMs for a Specific Task > *"You need to classify customer support tickets into 12 categories. You can use Claude Sonnet, GPT-4o-mini, or Llama 70B via Together AI. Pick the right model with proper evaluation."* --- ## Step 1 — Build the eval set 100 representative tickets with ground-truth labels: ```python eval_set = [ {"ticket": "My package hasn't arrived in 2 weeks", "label": "shipping"}, {"ticket": "I want to return these shoes", "label": "refund"}, {"ticket": "How do I use the loyalty rewards?", "label": "account"}, # ... 97 more ] # Save it import json with open("eval_set.json", "w") as f: json.dump(eval_set, f, indent=2) ``` Coverage: - All 12 categories represented. - Common cases (the "obvious" ones). - Edge cases (ambiguous tickets, multi-category). - Adversarial (prompt injection attempts, jokes, off-topic). 100 examples is the minimum for confident comparison. ## Step 2 — Define the task System prompt (same for all models): ```python CLASSIFY_PROMPT = '''Classify the customer support ticket into ONE of these 12 categories: - shipping (delivery questions, tracking, missing packages) - refund (returns, refund requests) - product (questions about features, fit, specs) - account (login, password, profile) - billing (charges, invoices, payment methods) - complaint (frustration without specific request) - technical (website/app bugs) - feature_request (suggestions for new features) - partnership (B2B inquiries) - press (media inquiries) - other (anything else) - spam (irrelevant or promotional content) Output ONLY the category label, nothing else. Ticket: {ticket} Category:''' ``` Same prompt for all candidates. Only the model changes. ## Step 3 — Run each model ```python import anthropic, openai, together claude = anthropic.Anthropic() gpt = openai.OpenAI() tg = together.Together() CANDIDATES = { "claude-sonnet-4-6": lambda prompt: claude.messages.create( model="claude-sonnet-4-6", messages=[{"role": "user", "content": prompt}], max_tokens=20, ).content[0].text.strip().lower(), "gpt-4o-mini": lambda prompt: gpt.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}], max_tokens=20, ).choices[0].message.content.strip().lower(), "llama-3-70b": lambda prompt: tg.chat.completions.create( model="meta-llama/Llama-3-70b-chat-hf", messages=[{"role": "user", "content": prompt}], max_tokens=20, ).choices[0].message.content.strip().lower(), } def run_eval(candidate_name, candidate_fn, eval_set): results = [] for ex in eval_set: prompt = CLASSIFY_PROMPT.format(ticket=ex["ticket"]) start = time.time() try: prediction = candidate_fn(prompt) except Exception as e: prediction = f"ERROR: {e}" latency_ms = (time.time() - start) * 1000 results.append({ "ticket": ex["ticket"], "expected": ex["label"], "predicted": prediction, "correct": prediction == ex["label"], "latency_ms": latency_ms, }) return results ``` ## Step 4 — Compute metrics ```python def summarize_results(results, candidate_name): correct = sum(r["correct"] for r in results) accuracy = correct / len(results) avg_latency = sum(r["latency_ms"] for r in results) / len(results) p95_latency = sorted(r["latency_ms"] for r in results)[int(len(results) * 0.95)] # Per-category accuracy from collections import defaultdict cat_correct, cat_total = defaultdict(int), defaultdict(int) for r in results: cat_total[r["expected"]] += 1 if r["correct"]: cat_correct[r["expected"]] += 1 cat_accuracy = {c: cat_correct[c] / cat_total[c] for c in cat_total} return { "candidate": candidate_name, "overall_accuracy": accuracy, "avg_latency_ms": avg_latency, "p95_latency_ms": p95_latency, "category_accuracy": cat_accuracy, } ``` ## Step 5 — Calculate cost Based on pricing per 1M tokens (Module 1): ```python PRICING = { "claude-sonnet-4-6": {"input": 3.00, "output": 15.00}, "gpt-4o-mini": {"input": 0.15, "output": 0.60}, "llama-3-70b": {"input": 0.90, "output": 0.90}, # Together AI } def estimate_cost_per_1M_tickets(results, candidate_name): # Rough estimate: 100 tokens input, 5 tokens output per request avg_input_tokens = 100 avg_output_tokens = 5 pricing = PRICING[candidate_name] cost_per_request = (avg_input_tokens * pricing["input"] / 1e6) + (avg_output_tokens * pricing["output"] / 1e6) cost_per_1M = cost_per_request * 1_000_000 return cost_per_1M ``` ## Step 6 — Compare results Suppose the eval produces: | Model | Accuracy | p95 latency | Cost / 1M requests | |---|---|---|---| | claude-sonnet-4-6 | 91% | 800ms | $375 | | gpt-4o-mini | 87% | 600ms | $18 | | llama-3-70b | 8