SUBMISSION: Case 5 — Distillation from Frontier LLM to Llama 3 with LoRA

# SUBMISSION: Case 5 — Model Distillation & Fine-Tuning: Domain Specialization from Frontier LLM to Fine-Tuned Llama 3 with LoRA > **Industry Context:** A high-volume ticket routing system classifies 5,000,000 tickets/month using Claude 3.5 Sonnet ($75,000/month). Distill this narrow domain task into an open-weight Llama 3.1 8B model fine-tuned via LoRA, reducing operating costs to under $2,000/month while maintaining 95%+ agreement with the frontier teacher. --- ## Deliverables 1. `dataset_curation.py`: Pipeline sampling 5,000 diverse production tickets, generating gold teacher labels via frontier LLM, and applying automated quality filters. 2. `train_lora.py`: PEFT/LoRA training script using Hugging Face `transformers`, `peft`, and `trl` (SFTTrainer), with 4-bit quantization (QLoRA) and target modules (`q_proj, v_proj`). 3. `eval_canary.py`: Evaluation suite proving >=95% agreement with the teacher model across a held-out test set, including a 5% canary shadow routing script. --- ## Starter Code Scaffold ```python from peft import LoraConfig, get_peft_model, TaskType lora_config = LoraConfig( task_type=TaskType.CAUSAL_LM, r=16, lora_alpha=32, lora_dropout=0.05, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"] ) # TODO: Initialize base model in 4-bit, attach LoRA adapter, train on curated dataset ``` --- ## 100-Point Evaluation Rubric - **Criterion 1: Distillation Dataset Preparation & Quality Gate (25 pts)** — Stratified sampling across 12 categories, teacher label verification, edge-case coverage. - **Criterion 2: LoRA / QLoRA Training Configuration (25 pts)** — Efficient rank selection ($r=16$), learning rate warmup, evaluation loss tracking, avoiding catastrophic forgetting. - **Criterion 3: Agreement Benchmarking (25 pts)** — Rigorous evaluation on held-out test data showing >=95% categorical agreement with the teacher model. - **Criterion 4: Production Deployment & Canary Routing (25 pts)** — Gradual rollout strategy (5% canary), automated latency telemetry, and fallback circuit breakers. --- ## Implementation Guide & Case Steps > *"You're running classification at 5M tickets/month using Claude Sonnet ($75K/month). Distill it into a fine-tuned Llama 3 8B that runs for $1.5K/month. Walk through the full process."* --- ## Step 1 — Validate the economics Current state: - 5M tickets/month at Claude Sonnet rates (~$0.015 per ticket) = $75K/month. Target state: - 5M tickets/month on fine-tuned Llama 3 8B via Together AI ($0.20 per 1M input + output) ≈ $1K-$2K/month. Savings: $73K/month. Fine-tuning cost: $100-500 one-time. Hosting (if Together): $0 (just per-token). Break-even: < 1 week. Worth doing. ## Step 2 — Generate the training data Distillation: use the strong model to label data; train the small model on those labels. ```python import anthropic claude = anthropic.Anthropic() def label_with_claude(ticket_text): response = claude.messages.create( model="claude-sonnet-4-6", messages=[{ "role": "user", "content": f'''Classify this customer support ticket into ONE category: - shipping, refund, product, account, billing, complaint, technical, feature_request, partnership, press, other, spam Ticket: {ticket_text} Category:''' }], max_tokens=20, ) return response.content[0].text.strip().lower() # Sample 10K real tickets, label each import json tickets = load_random_tickets(n=10000) training_data = [] for t in tickets: label = label_with_claude(t["text"]) training_data.append({"input": t["text"], "output": label}) if len(training_data) % 100 == 0: print(f"Labeled {len(training_data)}/{len(tickets)}") with open("training_data.jsonl", "w") as f: for ex in training_data: f.write(json.dumps(ex) + "\n") ``` Cost of labeling: 10K tickets × ~$0.015 = $150. Cheap. ## Step 3 — Audit a sample manually Don't trust Claude blindly. Sample 100-200 examples and verify labels: ```python import random sample = random.sample(training_data, 100) for ex in sample: print(f"INPUT: {ex['input'][:200]}") print(f"CLAUDE LABEL: {ex['output']}") your_label = input("Correct? (y/n/ ): ") if your_label == "n": new_label = input("Correct label: ") ex["output"] = new_label ``` If Claude is >95% accurate on your sample, training data is good enough. If <90%, improve the labeling prompt before continuing. ## Step 4 — Format for fine-tuning Most fine-tuning services expect a specific format: ```jsonl {"messages": [{"role": "user", "content": "Classify: ..."}, {"role": "assistant", "content": "shipping"}]} {"messages": [{"role": "user", "content": "Classify: ..."}, {"role": "assistant", "content": "refund"}]} ``` Save in JSONL format: ```python with open("train.jsonl", "w") as f: for ex in training_data: f.write(json.dumps({ "messages": [ {"role": "user", "content": f"Classify this ticket: {ex['input']}"