Case 5 — Design a Retraining Pipeline for a Fraud Model

Enterprise Scenario: An e-commerce marketplace's transaction fraud detection model experiences recurring seasonal drift and adversary evolution. Manual retraining by individual data scientists is slow, unrepeatable, and has led to silent regressions on high-value fraud cases. You must architect an automated retraining pipeline script (retrain_dag.py): support scheduled monthly execution and drift-triggered runs, partition rolling historical training windows, compete candidate models against the production incumbent under strict validation gates, and stage winning candidates for governed promotion.

Technical Requirements & Deliverables

Deliverable 1: Retraining DAG Architecture (retrain_dag.py)

• Script accepts --trigger-type ("scheduled" vs "drift_triggered") and --data-source.
• Incorporates data freshness validation: aborts pipeline if the incoming dataset has fewer than 10,000 labeled samples or is missing recent date windows.

Deliverable 2: Rolling Window Temporal Partitioning

• Ingest the latest 180 days of historical transactions for training.
• Ingest the subsequent 30 days for out-of-time validation, strictly preventing temporal lookahead leakage.

Deliverable 3: Multi-Objective Champion-Challenger Validation Gate

• Train the retrained candidate model using pinned hyperparameter configurations.
• Compare candidate against the active production champion (models:/fraud_detector/Production) on the identical out-of-time validation dataset.
• Candidate must pass all three gating criteria to qualify for promotion:
  1. Primary Metric: PR-AUC improves by $\ge 0.015$ over the champion.
  2. Guardrail Slice: High-value transaction recall ($\ge $500$) must not drop by more than $0.5%$.
  3. Latency Budget: Inference latency p95 is $\le 30 ext{ms}$.

Deliverable 4: Governed Registry Promotion & Webhook Notification

• If the candidate passes all gates:
  • Register as new version in MLflow Model Registry under fraud_detector.
  • Tag version with trigger_type, training_window, pr_auc_lift, and champion_version.
  • Transition model version to Staging.
  • Emit an automated notification payload for human sign-off prior to production traffic cutover.

Starter Code Scaffold

import argparse
import mlflow
import mlflow.sklearn
from datetime import datetime, timedelta
import pandas as pd
from sklearn.metrics import precision_recall_curve, auc, recall_score

def evaluate_gate(candidate_model, champion_model, val_df, feature_cols, target_col):
    X_val = val_df[feature_cols]
    y_val = val_df[target_col]
    
    # 1. PR-AUC Evaluation
    cand_probs = candidate_model.predict_proba(X_val)[:, 1]
    champ_probs = champion_model.predict_proba(X_val)[:, 1]
    
    p_c, r_c, _ = precision_recall_curve(y_val, cand_probs)
    p_ch, r_ch, _ = precision_recall_curve(y_val, champ_probs)
    cand_pr_auc = auc(r_c, p_c)
    champ_pr_auc = auc(r_ch, p_ch)
    
    pr_auc_passed = (cand_pr_auc - champ_pr_auc) >= 0.015
    
    # 2. High-Value Slice Recall
    high_val_mask = val_df["amount"] >= 500.0
    cand_slice_recall = recall_score(y_val[high_val_mask], cand_probs[high_val_mask] >= 0.5)
    champ_slice_recall = recall_score(y_val[high_val_mask], champ_probs[high_val_mask] >= 0.5)
    slice_passed = (champ_slice_recall - cand_slice_recall) <= 0.005
    
    return {
        "cand_pr_auc": cand_pr_auc,
        "champ_pr_auc": champ_pr_auc,
        "pr_auc_lift": cand_pr_auc - champ_pr_auc,
        "passed_gate": pr_auc_passed and slice_passed
    }

100-Point Graded Rubric

• Criteria 1: Pipeline Orchestration & Trigger Modality (25 pts)
  • 25 pts: Implements modular pipeline supporting both scheduled and drift-triggered execution; enforces sample volume assertions.
  • 15 pts: Script executes but does not support multiple trigger modalities or volume checks.
  • 0 pts: Script fails to execute or contains hardcoded static paths.
• Criteria 2: Temporal Window Partitioning (25 pts)
  • 25 pts: Strictly partitions rolling 180d training and 30d out-of-time validation windows; zero temporal leakage.
  • 15 pts: Uses random train-test splitting rather than temporal out-of-time partitions.
  • 0 pts: Demonstrates obvious data leakage between training and evaluation windows.
• Criteria 3: Multi-Objective Champion-Challenger Gate (25 pts)
  • 25 pts: Compares candidate directly against active production champion; asserts both aggregate PR-AUC lift and high-value slice recall protection.
  • 15 pts: Evaluates candidate against static metric threshold without direct champion comparison.
  • 0 pts: Auto-promotes candidate without validation gating.
• Criteria 4: Registry Governance & Notification (25 pts)
  • 25 pts: Stages approved models in MLflow Model Registry with complete lineage tags; outputs structured promotion notification.
  • 15 pts: Registers model in MLflow but omits lineage metadata tags.
  • 0 pts: Fails to interact with the Model Registry.