LLMs & Generative AI for Practitioners
Back to RAG, Fine-tuning, and Evaluation

SUBMISSION: Case 5 — Distillation from Frontier LLM to Llama 3 with LoRA

Distill Claude Opus into a fine-tuned Llama 8B for production cost savings.

114 minutesProject submission
🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Assignment Instructions

Industry Context: A high-volume ticket routing system classifies 5,000,000 tickets/month using Claude 3.5 Sonnet ($75,000/month). Distill this narrow domain task into an open-weight Llama 3.1 8B model fine-tuned via LoRA, reducing operating costs to under $2,000/month while maintaining 95%+ agreement with the frontier teacher.

Deliverables

  1. dataset_curation.py: Pipeline sampling 5,000 diverse production tickets, generating gold teacher labels via frontier LLM, and applying automated quality filters.
  2. train_lora.py: PEFT/LoRA training script using Hugging Face transformers, peft, and trl (SFTTrainer), with 4-bit quantization (QLoRA) and target modules (q_proj, v_proj).
  3. eval_canary.py: Evaluation suite proving >=95% agreement with the teacher model across a held-out test set, including a 5% canary shadow routing script.

Starter Code Scaffold

from peft import LoraConfig, get_peft_model, TaskType

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"]
)

# TODO: Initialize base model in 4-bit, attach LoRA adapter, train on curated dataset

100-Point Evaluation Rubric

  • Criterion 1: Distillation Dataset Preparation & Quality Gate (25 pts) — Stratified sampling across 12 categories, teacher label verification, edge-case coverage.
  • Criterion 2: LoRA / QLoRA Training Configuration (25 pts) — Efficient rank selection ($r=16$), learning rate warmup, evaluation loss tracking, avoiding catastrophic forgetting.
  • Criterion 3: Agreement Benchmarking (25 pts) — Rigorous evaluation on held-out test data showing >=95% categorical agreement with the teacher model.
  • Criterion 4: Production Deployment & Canary Routing (25 pts) — Gradual rollout strategy (5% canary), automated latency telemetry, and fallback circuit breakers.

Implementation Guide & Case Steps

"You're running classification at 5M tickets/month using Claude Sonnet ($75K/month). Distill it into a fine-tuned Llama 3 8B that runs for $1.5K/month. Walk through the full process."

Step 1 — Validate the economics

Current state:

  • 5M tickets/month at Claude Sonnet rates (~$0.015 per ticket) = $75K/month.

Target state:

  • 5M tickets/month on fine-tuned Llama 3 8B via Together AI ($0.20 per 1M input + output) ≈ $1K-$2K/month.

Savings: $73K/month. Fine-tuning cost: $100-500 one-time. Hosting (if Together): $0 (just per-token).

Break-even: < 1 week. Worth doing.

Step 2 — Generate the training data

Distillation: use the strong model to label data; train the small model on those labels.

import anthropic
claude = anthropic.Anthropic()

def label_with_claude(ticket_text):
    response = claude.messages.create(
        model="claude-sonnet-4-6",
        messages=[{
            "role": "user",
            "content": f'''Classify this customer support ticket into ONE category:
- shipping, refund, product, account, billing, complaint, technical, feature_request, partnership, press, other, spam

Ticket: {ticket_text}

Category:'''
        }],
        max_tokens=20,
    )
    return response.content[0].text.strip().lower()

# Sample 10K real tickets, label each
import json

tickets = load_random_tickets(n=10000)
training_data = []
for t in tickets:
    label = label_with_claude(t["text"])
    training_data.append({"input": t["text"], "output": label})
    if len(training_data) % 100 == 0:
        print(f"Labeled {len(training_data)}/{len(tickets)}")

with open("training_data.jsonl", "w") as f:
    for ex in training_data:
        f.write(json.dumps(ex) + "\n")

Cost of labeling: 10K tickets × ~$0.015 = $150. Cheap.

Step 3 — Audit a sample manually

Don't trust Claude blindly. Sample 100-200 examples and verify labels:

import random
sample = random.sample(training_data, 100)
for ex in sample:
    print(f"INPUT: {ex['input'][:200]}")
    print(f"CLAUDE LABEL: {ex['output']}")
    your_label = input("Correct? (y/n/<label>): ")
    if your_label == "n":
        new_label = input("Correct label: ")
        ex["output"] = new_label

If Claude is >95% accurate on your sample, training data is good enough. If <90%, improve the labeling prompt before continuing.

Step 4 — Format for fine-tuning

Most fine-tuning services expect a specific format:

{"messages": [{"role": "user", "content": "Classify: ..."}, {"role": "assistant", "content": "shipping"}]}
{"messages": [{"role": "user", "content": "Classify: ..."}, {"role": "assistant", "content": "refund"}]}

Save in JSONL format:

with open("train.jsonl", "w") as f:
    for ex in training_data:
        f.write(json.dumps({
            "messages": [
                {"role": "user", "content": f"Classify this ticket: {ex['input']}"},
                {"role": "assistant", "content": ex["output"]}
            ]
        }) + "\n")

Split into train (90%) and validation (10%).

Step 5 — Run the fine-tune

Using Together AI (open model fine-tuning):

import together

together.api_key = os.environ["TOGETHER_API_KEY"]

# Upload training data
resp = together.Files.upload(file="train.jsonl", check=True)
train_file_id = resp["id"]

resp = together.Files.upload(file="val.jsonl", check=True)
val_file_id = resp["id"]

# Start fine-tuning
job = together.Finetune.create(
    training_file=train_file_id,
    validation_file=val_file_id,
    model="meta-llama/Llama-3-8b-chat-hf",
    n_epochs=3,
    lora=True,  # LoRA for efficiency
    learning_rate=1e-5,
)
print(f"Fine-tune started: {job.id}")

Wait for completion (1-4 hours typical for 10K examples). Together returns a fine-tuned model name like llama-3-8b-acme-classifier.

Step 6 — Evaluate the fine-tuned model

Use the SAME eval set as before (100 manually-labeled tickets):

def eval_finetuned():
    model = "llama-3-8b-acme-classifier"
    results = []
    for ex in eval_set:
        response = together.Chat.completions.create(
            model=model,
            messages=[
                {"role": "user", "content": f"Classify this ticket: {ex['ticket']}"}
            ],
            max_tokens=20,
        )
        prediction = response.choices[0].message.content.strip().lower()
        results.append({
            "ticket": ex["ticket"],
            "expected": ex["label"],
            "predicted": prediction,
            "correct": prediction == ex["label"],
        })
    return sum(r["correct"] for r in results) / len(results)

print(f"Fine-tuned accuracy: {eval_finetuned()}")
print(f"Original Claude Sonnet accuracy: 91%")

Typical result: fine-tuned 8B model gets 85-90% accuracy. Close to but not equal to Claude Sonnet's 91%.

If acceptable, proceed. If not, iterate:

  • Add more training data.
  • Increase epochs.
  • Improve labeling quality (better Claude prompt or human review).

Step 7 — Deploy the fine-tuned model

# In production
def classify_ticket(ticket_text):
    response = together.Chat.completions.create(
        model="llama-3-8b-acme-classifier",
        messages=[
            {"role": "user", "content": f"Classify this ticket: {ticket_text}"}
        ],
        max_tokens=20,
        temperature=0,  # deterministic
    )
    return response.choices[0].message.content.strip().lower()

Rollout strategy:

  • Week 1: 5% canary, A/B against Claude Sonnet (shadow mode).
  • Week 2: 50/50 split, compare metrics.
  • Week 3: 100% on fine-tuned model, holdout 5% on Claude for ongoing comparison.

Step 8 — Monitor in production

-- Compare quality across the holdout split
SELECT
  model_version,
  COUNT(*) AS requests,
  AVG(latency_ms) AS avg_latency,
  AVG(cost) AS avg_cost,
  SUM(cost) AS total_cost,
  -- Match against eventually-labeled ground truth
  AVG(CASE WHEN predicted = true_label THEN 1.0 ELSE 0.0 END) AS accuracy
FROM ticket_classifications c
LEFT JOIN ticket_labels l ON l.ticket_id = c.ticket_id
WHERE c.served_at >= NOW() - INTERVAL '7 days'
GROUP BY model_version;

Track over time. If fine-tuned model's accuracy degrades (data distribution shift), retrain.

Step 9 — Plan retraining cadence

Domain data shifts. Quarterly retraining is a reasonable default:

# .github/workflows/quarterly_retrain.yml
on:
  schedule:
    - cron: '0 0 1 1,4,7,10 *'  # 1st of Jan, Apr, Jul, Oct

jobs:
  retrain:
    runs-on: ubuntu-latest
    steps:
      - run: python scripts/retrain_classifier.py
      - run: python scripts/evaluate_new_classifier.py
      - run: python scripts/promote_if_better.py

If retrained model is meaningfully better on eval, promote it. If not, keep current.

Result

  • Cost: $75K/month → $1.5K/month (~50x reduction).
  • Accuracy: 91% → 88% (3 points lower; acceptable for this use case).
  • Latency: comparable (Together AI's 8B model is fast).
  • Quality monitoring: ongoing via 5% holdout on Claude.

What this case teaches

  1. Distillation: frontier model labels → smaller model trains → cheap production.
  2. Manual data audit is essential — Claude isn't perfect.
  3. LoRA + Together = cheap, fast fine-tuning.
  4. Maintain a holdout on the original model for ongoing comparison.
  5. Quarterly retraining for distribution shift.

Self-check MCQs

Checking your access…
← Previous: Quiz: Fine-Tuning & Model DistillationNext: Evaluating LLM Outputs — The Hard Problem