Fine-Tuning Discipline (Freeze Schedules, LR, Leakage, Eval)

This lesson on Fine-Tuning Discipline (Freeze Schedules, LR, Leakage, Eval) is hands-on and example-driven. You will design robust fine-tuning pipelines by systematically freezing backbone layers, applying discriminative learning rates, and preventing data leakage across dataset splits. You will implement a shared evaluation protocol that measures downstream performance without corrupting pretrained representations.

What You'll Be Able To Do

• Implement freeze and gradual unfreeze schedules across PyTorch model parameter groups.
• Configure discriminative learning rates to protect early feature extractors while training classification heads.
• Prevent target leakage and normalization stat contamination across training and validation splits.
• Write a standardized evaluation protocol shared across fine-tuning benchmark mini-labs.

Detailed Concept Walkthrough

1. Layer Freezing and Gradual Unfreezing Schedules

Freezing prevents pretrained backbone weights from being corrupted by large, random gradient updates originating from newly initialized classifier heads. Gradual unfreezing slowly opens earlier layers as top-level representations stabilize.

• Mechanism: Setting param.requires_grad = False removes tensors from autograd graph tracking, skipping gradient computation and optimizer updates during backpropagation.
• Under the Hood: Freezing reduces GPU memory consumption and accelerates backward passes because the computational graph terminates before computing gradients for frozen parameters.
• Execution Flow: Train the randomly initialized classification head first for several epochs, then unfreeze the top backbone layers, and finally unfreeze deeper feature layers at lower learning rates.

import torch.nn as nn

def set_freeze_schedule(model: nn.Module, freeze_backbone: bool = True):
    # Freeze or unfreeze feature extractor parameters
    for param in model.backbone.parameters():
        param.requires_grad = not freeze_backbone
    # Ensure classification head remains trainable
    for param in model.classifier.parameters():
        param.requires_grad = True

Key Takeaway: Always stabilize newly initialized heads on frozen backbones before fine-tuning pretrained layers.

2. Differential Learning Rates and Optimizers

Lower layers capture generic, transferrable representations that require minimal adjustment, whereas task-specific top layers require faster adaptation.

• Mechanism: PyTorch optimizers accept parameter dictionaries with distinct learning rates, weight decays, and scheduling rules per layer group.
• Syntax Rule: Pass a list of parameter group dicts to torch.optim.AdamW or SGD, explicitly defining params and lr for each group.
• Best Practice: Assign backbone layers an order of magnitude lower learning rate (e.g., 1e-5 to 1e-4) compared to the classifier head (e.g., 1e-3).

import torch.optim as optim

optimizer = optim.AdamW([
    {'params': model.backbone.parameters(), 'lr': 1e-5, 'weight_decay': 0.01},
    {'params': model.classifier.parameters(), 'lr': 1e-3, 'weight_decay': 0.0}
])

Key Takeaway: Assign progressively smaller learning rates to earlier layers to prevent catastrophic forgetting of general features.

3. Data Leakage Prevention Across Splits

Fine-tuning pipelines must isolate training distributions from validation and test sets during both preprocessing and augmentation steps.

• Mechanism: Compute dataset statistics (e.g., mean and standard deviation) solely on the training split, or strictly adopt the exact pretrained model's preset normalization values.
• Under the Hood: Applying stochastic data augmentations (e.g., random crops, flips) to validation data introduces distribution variance and distorts deterministic performance assessment.
• Best Practice: Separate transformation pipelines into a stochastic training pipeline and a deterministic validation/test evaluation pipeline.

from torchvision import transforms

# Pretrained normalization values (e.g., ImageNet)
NORM = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])

train_tfms = transforms.Compose([transforms.RandomResizedCrop(224), transforms.RandomHorizontalFlip(), transforms.ToTensor(), NORM])
eval_tfms = transforms.Compose([transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), NORM])

Key Takeaway: Never compute normalization statistics over test data or apply stochastic augmentations during evaluation passes.

4. Standardized Evaluation Protocols

A unified evaluation protocol guarantees fair, reproducible comparisons between different fine-tuning checkpoints and baseline configurations.

• Mechanism: Switch models to model.eval() and wrap inference in torch.no_grad() to disable dropout and freeze batch normalization running statistics.
• Execution Flow: Accumulate batch predictions and ground-truth targets into flat tensors before computing global accuracy, precision, recall, or F1 scores.
• Best Practice: Compute and record both loss and task-specific classification metrics across all mini-lab iterations using identical metric routines.

import torch

def evaluate_protocol(model, dataloader, criterion, device):
    model.eval()
    total_loss, correct, total = 0.0, 0, 0
    with torch.no_grad():
        for inputs, targets in dataloader:
            inputs, targets = inputs.to(device), targets.to(device)
            outputs = model(inputs)
            total_loss += criterion(outputs, targets).item() * inputs.size(0)
            preds = outputs.argmax(dim=1)
            correct += (preds == targets).sum().item()
            total += targets.size(0)
    return total_loss / total, correct / total

Key Takeaway: Always execute validation under torch.no_grad() and model.eval() to prevent tracking graphs and updating batch statistics.

Topics Covered in Fine-Tuning Discipline (Freeze Schedules, LR, Leakage, Eval)

1. Fine-Tuning Challenges (0:00 - 1:30) — Identifies risks of catastrophic forgetting and gradient shock during transfer learning.
2. Freeze Schedules & Gradual Unfreezing (1:30 - 3:15) — Demonstrates disabling gradients on pretrained backbones before selectively opening top layers.
3. Differential Learning Rates (3:15 - 4:45) — Sets up parameter group dictionaries with distinct learning rates inside PyTorch optimizers.
4. Preventing Data Leakage (4:45 - 6:15) — Separates training augmentations from deterministic validation transforms and dataset statistics.
5. Shared Evaluation Protocol (6:15 - 8:00) — Constructs the reusable evaluation harness using eval mode and no-grad contexts for mini-labs.

DL Basics Cheat Sheet

• param.requires_grad = False — Disables autograd gradient tracking for targeted model parameters

  for p in model.backbone.parameters(): p.requires_grad = False
  

• optim.Optimizer([{'params': ...}]) — Configures layer-specific learning rates across parameter groups

  optim.AdamW([{'params': model.feat.parameters(), 'lr': 1e-4}, {'params': model.fc.parameters(), 'lr': 1e-3}])
  

• model.eval() — Sets dropout and batch normalization layers to inference mode

  model.eval()
  

• with torch.no_grad(): — Deactivates autograd memory overhead during validation loops

  with torch.no_grad(): out = model(x_val)
  

• filter(lambda p: p.requires_grad, model.parameters()) — Passes only active trainable parameters to the optimizer

  optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-3)
  

Comparison Table

Common Pitfalls

• Mistake: Passing frozen parameters to an optimizer that updates all parameters uniformly. Avoid: Filter parameters with requires_grad or pass explicit parameter groups.
• Mistake: Forgetting model.eval() before running validation passes. Avoid: Always set model.eval() to freeze dropout masks and running batch norm statistics.
• Mistake: Fitting image normalization statistics on combined training and test splits. Avoid: Fit transforms strictly on training data or use preset backbone normalization statistics.

FAQs

• Why does training a head without freezing the backbone lead to catastrophic forgetting? Randomly initialized head weights produce massive initial gradients that rapidly destroy calibrated feature representations in earlier layers.
• How does torch.no_grad() differ from model.eval()? torch.no_grad() disables autograd tape recording to save memory, whereas model.eval() changes module runtime behaviors like dropout and batchnorm.
• When should gradual unfreezing be preferred over full fine-tuning immediately? Gradual unfreezing is preferred when working with small downstream datasets to prevent overfitting and gradient shock.