SUBMISSION: Case 1 — Multi-Model Evaluation on Ticket Classification

# SUBMISSION: Case 1 — Model Selection & Benchmarking: Multi-Model Evaluation on Real-World Ticket Classification > **Industry Context:** An enterprise SaaS customer support desk handles 250,000 inbound tickets monthly across 12 support categories. You must benchmark candidate LLM architectures (Claude 3.5 Sonnet, GPT-4o-mini, and Llama 3.1 70B) against a 100-ticket ground-truth dataset to select the optimal model balancing accuracy, latency, and budget. --- ## Deliverables 1. `eval_benchmark.py`: Standardized evaluation pipeline measuring accuracy, per-class F1, and latency across all three candidate models with temperature=0 and JSON schema output. 2. `benchmark_results.json`: Complete confusion matrix, per-category breakdown, and unit cost model at 250,000 tickets/month. 3. `model_selection_memo.md`: 2-page executive engineering recommendation detailing accuracy vs cost trade-offs and recommending the primary + fallback model routing strategy. --- ## Starter Code Scaffold ```python import json import time from typing import Dict, List, Any from pydantic import BaseModel, Field class TicketClassification(BaseModel): category: str = Field(description="One of: billing, auth, bug, feature_request, sla_breach, account, api, data_export, security, hardware, onboarding, other") confidence: float = Field(ge=0.0, le=1.0) reasoning: str CATEGORIES = [ "billing", "auth", "bug", "feature_request", "sla_breach", "account", "api", "data_export", "security", "hardware", "onboarding", "other" ] def run_evaluation(eval_set_path: str, model_name: str) -> Dict[str, Any]: """Run standardized evaluation loop over ground-truth tickets.""" # TODO: Ingest tickets, invoke model endpoint with response_format, calculate accuracy and latency pass ``` --- ## 100-Point Evaluation Rubric - **Criterion 1: Standardized Evaluation Methodology (25 pts)** — Uniform prompt framing, strict schema parsing, zero prompt leakage, deterministic sampling (`temperature=0.0`). - **Criterion 2: Granular Per-Category & Ambiguity Scoring (25 pts)** — Full confusion matrix, macro-F1, analysis of boundary errors (e.g., billing disputes vs SLA breaches). - **Criterion 3: Unit Economics & Latency Telemetry (25 pts)** — Accurate input/output token cost modeling at 250K monthly volume, P50/P95 latency measurement. - **Criterion 4: Architectural Decision Memo (25 pts)** — Rigorous executive synthesis, justifiable model selection, and practical fallback routing architecture. --- ## Implementation Guide & Case Steps > *"You need to classify customer support tickets into 12 categories. You can use Claude Sonnet, GPT-4o-mini, or Llama 70B via Together AI. Pick the right model with proper evaluation."* --- ## Step 1 — Build the eval set 100 representative tickets with ground-truth labels: ```python eval_set = [ {"ticket": "My package hasn't arrived in 2 weeks", "label": "shipping"}, {"ticket": "I want to return these shoes", "label": "refund"}, {"ticket": "How do I use the loyalty rewards?", "label": "account"}, # ... 97 more ] # Save it import json with open("eval_set.json", "w") as f: json.dump(eval_set, f, indent=2) ``` Coverage: - All 12 categories represented. - Common cases (the "obvious" ones). - Edge cases (ambiguous tickets, multi-category). - Adversarial (prompt injection attempts, jokes, off-topic). 100 examples is the minimum for confident comparison. ## Step 2 — Define the task System prompt (same for all models): ```python CLASSIFY_PROMPT = '''Classify the customer support ticket into ONE of these 12 categories: - shipping (delivery questions, tracking, missing packages) - refund (returns, refund requests) - product (questions about features, fit, specs) - account (login, password, profile) - billing (charges, invoices, payment methods) - complaint (frustration without specific request) - technical (website/app bugs) - feature_request (suggestions for new features) - partnership (B2B inquiries) - press (media inquiries) - other (anything else) - spam (irrelevant or promotional content) Output ONLY the category label, nothing else. Ticket: {ticket} Category:''' ``` Same prompt for all candidates. Only the model changes. ## Step 3 — Run each model ```python import anthropic, openai, together claude = anthropic.Anthropic() gpt = openai.OpenAI() tg = together.Together() CANDIDATES = { "claude-sonnet-4-6": lambda prompt: claude.messages.create( model="claude-sonnet-4-6", messages=[{"role": "user", "content": prompt}], max_tokens=20, ).content[0].text.strip().lower(), "gpt-4o-mini": lambda prompt: gpt.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}], max_tokens=20, ).choices[0].message.content.strip().lower(), "llama-3-70b": lambda prompt: tg.chat.completions.create( model="meta-llama/Llama-3-70b-chat-hf", messages=[{"role": "user", "content": pro