Case 6 — Fine-tune a Small Model for Domain Specialization

# Case 6 — Fine-tune a Small Model for Domain Specialization > *"You're running classification at 5M tickets/month using Claude Sonnet ($75K/month). Distill it into a fine-tuned Llama 3 8B that runs for $1.5K/month. Walk through the full process."* --- ## Step 1 — Validate the economics Current state: - 5M tickets/month at Claude Sonnet rates (~$0.015 per ticket) = $75K/month. Target state: - 5M tickets/month on fine-tuned Llama 3 8B via Together AI ($0.20 per 1M input + output) ≈ $1K-$2K/month. Savings: $73K/month. Fine-tuning cost: $100-500 one-time. Hosting (if Together): $0 (just per-token). Break-even: < 1 week. Worth doing. ## Step 2 — Generate the training data Distillation: use the strong model to label data; train the small model on those labels. ```python import anthropic claude = anthropic.Anthropic() def label_with_claude(ticket_text): response = claude.messages.create( model="claude-sonnet-4-6", messages=[{ "role": "user", "content": f'''Classify this customer support ticket into ONE category: - shipping, refund, product, account, billing, complaint, technical, feature_request, partnership, press, other, spam Ticket: {ticket_text} Category:''' }], max_tokens=20, ) return response.content[0].text.strip().lower() # Sample 10K real tickets, label each import json tickets = load_random_tickets(n=10000) training_data = [] for t in tickets: label = label_with_claude(t["text"]) training_data.append({"input": t["text"], "output": label}) if len(training_data) % 100 == 0: print(f"Labeled {len(training_data)}/{len(tickets)}") with open("training_data.jsonl", "w") as f: for ex in training_data: f.write(json.dumps(ex) + "\n") ``` Cost of labeling: 10K tickets × ~$0.015 = $150. Cheap. ## Step 3 — Audit a sample manually Don't trust Claude blindly. Sample 100-200 examples and verify labels: ```python import random sample = random.sample(training_data, 100) for ex in sample: print(f"INPUT: {ex['input'][:200]}") print(f"CLAUDE LABEL: {ex['output']}") your_label = input("Correct? (y/n/ ): ") if your_label == "n": new_label = input("Correct label: ") ex["output"] = new_label ``` If Claude is >95% accurate on your sample, training data is good enough. If <90%, improve the labeling prompt before continuing. ## Step 4 — Format for fine-tuning Most fine-tuning services expect a specific format: ```jsonl {"messages": [{"role": "user", "content": "Classify: ..."}, {"role": "assistant", "content": "shipping"}]} {"messages": [{"role": "user", "content": "Classify: ..."}, {"role": "assistant", "content": "refund"}]} ``` Save in JSONL format: ```python with open("train.jsonl", "w") as f: for ex in training_data: f.write(json.dumps({ "messages": [ {"role": "user", "content": f"Classify this ticket: {ex['input']}"}, {"role": "assistant", "content": ex["output"]} ] }) + "\n") ``` Split into train (90%) and validation (10%). ## Step 5 — Run the fine-tune Using Together AI (open model fine-tuning): ```python import together together.api_key = os.environ["TOGETHER_API_KEY"] # Upload training data resp = together.Files.upload(file="train.jsonl", check=True) train_file_id = resp["id"] resp = together.Files.upload(file="val.jsonl", check=True) val_file_id = resp["id"] # Start fine-tuning job = together.Finetune.create( training_file=train_file_id, validation_file=val_file_id, model="meta-llama/Llama-3-8b-chat-hf", n_epochs=3, lora=True, # LoRA for efficiency learning_rate=1e-5, ) print(f"Fine-tune started: {job.id}") ``` Wait for completion (1-4 hours typical for 10K examples). Together returns a fine-tuned model name like `llama-3-8b-acme-classifier`. ## Step 6 — Evaluate the fine-tuned model Use the SAME eval set as before (100 manually-labeled tickets): ```python def eval_finetuned(): model = "llama-3-8b-acme-classifier" results = [] for ex in eval_set: response = together.Chat.completions.create( model=model, messages=[ {"role": "user", "content": f"Classify this ticket: {ex['ticket']}"} ], max_tokens=20, ) prediction = response.choices[0].message.content.strip().lower() results.append({ "ticket": ex["ticket"], "expected": ex["label"], "predicted": prediction, "correct": prediction == ex["label"], }) return sum(r["correct"] for r in results) / len(results) print(f"Fine-tuned accuracy: {eval_finetuned()}") print(f"Original Claude Sonnet accuracy: 91%") ``` Typical result: fine-tuned 8B model gets 85-90% accuracy. Close to but not equal to Claude Sonnet's 91%. If acceptable, proceed. If not, iterate: - Add more training data. - Increase epochs. - Improve labeling quality (better Claude pro