SUBMISSION: Case 1 — Multi-Model Evaluation on Ticket Classification

Industry Context: An enterprise SaaS customer support desk handles 250,000 inbound tickets monthly across 12 support categories. You must benchmark candidate LLM architectures (Claude 3.5 Sonnet, GPT-4o-mini, and Llama 3.1 70B) against a 100-ticket ground-truth dataset to select the optimal model balancing accuracy, latency, and budget.

Deliverables

eval_benchmark.py: Standardized evaluation pipeline measuring accuracy, per-class F1, and latency across all three candidate models with temperature=0 and JSON schema output.
benchmark_results.json: Complete confusion matrix, per-category breakdown, and unit cost model at 250,000 tickets/month.
model_selection_memo.md: 2-page executive engineering recommendation detailing accuracy vs cost trade-offs and recommending the primary + fallback model routing strategy.

Starter Code Scaffold

import json
import time
from typing import Dict, List, Any
from pydantic import BaseModel, Field

class TicketClassification(BaseModel):
    category: str = Field(description="One of: billing, auth, bug, feature_request, sla_breach, account, api, data_export, security, hardware, onboarding, other")
    confidence: float = Field(ge=0.0, le=1.0)
    reasoning: str

CATEGORIES = [
    "billing", "auth", "bug", "feature_request", "sla_breach",
    "account", "api", "data_export", "security", "hardware", "onboarding", "other"
]

def run_evaluation(eval_set_path: str, model_name: str) -> Dict[str, Any]:
    """Run standardized evaluation loop over ground-truth tickets."""
    # TODO: Ingest tickets, invoke model endpoint with response_format, calculate accuracy and latency
    pass

100-Point Evaluation Rubric

Criterion 1: Standardized Evaluation Methodology (25 pts) — Uniform prompt framing, strict schema parsing, zero prompt leakage, deterministic sampling (temperature=0.0).
Criterion 2: Granular Per-Category & Ambiguity Scoring (25 pts) — Full confusion matrix, macro-F1, analysis of boundary errors (e.g., billing disputes vs SLA breaches).
Criterion 3: Unit Economics & Latency Telemetry (25 pts) — Accurate input/output token cost modeling at 250K monthly volume, P50/P95 latency measurement.
Criterion 4: Architectural Decision Memo (25 pts) — Rigorous executive synthesis, justifiable model selection, and practical fallback routing architecture.

Implementation Guide & Case Steps

"You need to classify customer support tickets into 12 categories. You can use Claude Sonnet, GPT-4o-mini, or Llama 70B via Together AI. Pick the right model with proper evaluation."

Step 1 — Build the eval set

100 representative tickets with ground-truth labels:

eval_set = [
    {"ticket": "My package hasn't arrived in 2 weeks", "label": "shipping"},
    {"ticket": "I want to return these shoes", "label": "refund"},
    {"ticket": "How do I use the loyalty rewards?", "label": "account"},
    # ... 97 more
]

# Save it
import json
with open("eval_set.json", "w") as f:
    json.dump(eval_set, f, indent=2)

Coverage:

All 12 categories represented.
Common cases (the "obvious" ones).
Edge cases (ambiguous tickets, multi-category).
Adversarial (prompt injection attempts, jokes, off-topic).

100 examples is the minimum for confident comparison.

Step 2 — Define the task

System prompt (same for all models):

CLASSIFY_PROMPT = '''Classify the customer support ticket into ONE of these 12 categories:
- shipping (delivery questions, tracking, missing packages)
- refund (returns, refund requests)
- product (questions about features, fit, specs)
- account (login, password, profile)
- billing (charges, invoices, payment methods)
- complaint (frustration without specific request)
- technical (website/app bugs)
- feature_request (suggestions for new features)
- partnership (B2B inquiries)
- press (media inquiries)
- other (anything else)
- spam (irrelevant or promotional content)

Output ONLY the category label, nothing else.

Ticket: {ticket}
Category:'''

Same prompt for all candidates. Only the model changes.

Step 3 — Run each model

import anthropic, openai, together

claude = anthropic.Anthropic()
gpt = openai.OpenAI()
tg = together.Together()

CANDIDATES = {
    "claude-sonnet-4-6": lambda prompt: claude.messages.create(
        model="claude-sonnet-4-6",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=20,
    ).content[0].text.strip().lower(),

    "gpt-4o-mini": lambda prompt: gpt.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=20,
    ).choices[0].message.content.strip().lower(),

    "llama-3-70b": lambda prompt: tg.chat.completions.create(
        model="meta-llama/Llama-3-70b-chat-hf",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=20,
    ).choices[0].message.content.strip().lower(),
}

def run_eval(candidate_name, candidate_fn, eval_set):
    results = []
    for ex in eval_set:
        prompt = CLASSIFY_PROMPT.format(ticket=ex["ticket"])
        start = time.time()
        try:
            prediction = candidate_fn(prompt)
        except Exception as e:
            prediction = f"ERROR: {e}"
        latency_ms = (time.time() - start) * 1000

        results.append({
            "ticket": ex["ticket"],
            "expected": ex["label"],
            "predicted": prediction,
            "correct": prediction == ex["label"],
            "latency_ms": latency_ms,
        })
    return results

Step 4 — Compute metrics

def summarize_results(results, candidate_name):
    correct = sum(r["correct"] for r in results)
    accuracy = correct / len(results)
    avg_latency = sum(r["latency_ms"] for r in results) / len(results)
    p95_latency = sorted(r["latency_ms"] for r in results)[int(len(results) * 0.95)]

    # Per-category accuracy
    from collections import defaultdict
    cat_correct, cat_total = defaultdict(int), defaultdict(int)
    for r in results:
        cat_total[r["expected"]] += 1
        if r["correct"]:
            cat_correct[r["expected"]] += 1
    cat_accuracy = {c: cat_correct[c] / cat_total[c] for c in cat_total}

    return {
        "candidate": candidate_name,
        "overall_accuracy": accuracy,
        "avg_latency_ms": avg_latency,
        "p95_latency_ms": p95_latency,
        "category_accuracy": cat_accuracy,
    }

Step 5 — Calculate cost

Based on pricing per 1M tokens (Module 1):

PRICING = {
    "claude-sonnet-4-6": {"input": 3.00, "output": 15.00},
    "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    "llama-3-70b": {"input": 0.90, "output": 0.90},  # Together AI
}

def estimate_cost_per_1M_tickets(results, candidate_name):
    # Rough estimate: 100 tokens input, 5 tokens output per request
    avg_input_tokens = 100
    avg_output_tokens = 5

    pricing = PRICING[candidate_name]
    cost_per_request = (avg_input_tokens * pricing["input"] / 1e6) + (avg_output_tokens * pricing["output"] / 1e6)
    cost_per_1M = cost_per_request * 1_000_000

    return cost_per_1M

Step 6 — Compare results

Suppose the eval produces:

Model	Accuracy	p95 latency	Cost / 1M requests
claude-sonnet-4-6	91%	800ms	$375
gpt-4o-mini	87%	600ms	$18
llama-3-70b	85%	1200ms	$95

Decisions:

If 91% vs 87% matters and budget allows: Claude Sonnet.
If $18 vs $375 matters (high-volume) and 4% accuracy is acceptable: GPT-4o-mini.
Llama 70B: middle of the pack on cost and quality; harder to justify here.

Step 7 — Drill into errors

Categories where one model dominates:

def category_comparison(all_results):
    comparison = {}
    for candidate, results in all_results.items():
        from collections import defaultdict
        cat_correct = defaultdict(int)
        cat_total = defaultdict(int)
        for r in results:
            cat_total[r["expected"]] += 1
            if r["correct"]:
                cat_correct[r["expected"]] += 1
        comparison[candidate] = {c: cat_correct[c] / cat_total[c] for c in cat_total}
    return comparison

Maybe GPT-4o-mini is 95% on common categories but 60% on the "feature_request" category. Claude Sonnet is 90% on both. Decision changes.

Step 8 — Document and decide

Report:

Recommended: GPT-4o-mini.

Decision factors:

At 100K tickets/month, cost is $1.80/month (Claude Sonnet would be $37.50/month).

87% accuracy is acceptable for this use case (errors auto-route to "other" → human review).

GPT-4o-mini's weak categories (feature_request) are low-volume.

Trade-offs documented. If quality requirements tighten or volume changes, re-evaluate.

What this case teaches

Benchmark on your data, not vendor benchmarks.
Same prompt across candidates for apples-to-apples.
Multiple metrics: accuracy, latency, cost.
Per-category breakdown reveals strengths/weaknesses hidden in aggregates.
Decision documented with rationale.