Case 5 — Design a Retraining Pipeline for a Fraud Model

# Case 5 — Design a Retraining Pipeline for a Fraud Model > **Enterprise Scenario:** An e-commerce marketplace's transaction fraud detection model experiences recurring seasonal drift and adversary evolution. Manual retraining by individual data scientists is slow, unrepeatable, and has led to silent regressions on high-value fraud cases. You must architect an automated retraining pipeline script (`retrain_dag.py`): support scheduled monthly execution and drift-triggered runs, partition rolling historical training windows, compete candidate models against the production incumbent under strict validation gates, and stage winning candidates for governed promotion. --- ## Technical Requirements & Deliverables ### Deliverable 1: Retraining DAG Architecture (`retrain_dag.py`) - Script accepts `--trigger-type` (`"scheduled"` vs `"drift_triggered"`) and `--data-source`. - Incorporates data freshness validation: aborts pipeline if the incoming dataset has fewer than 10,000 labeled samples or is missing recent date windows. ### Deliverable 2: Rolling Window Temporal Partitioning - Ingest the latest 180 days of historical transactions for training. - Ingest the subsequent 30 days for out-of-time validation, strictly preventing temporal lookahead leakage. ### Deliverable 3: Multi-Objective Champion-Challenger Validation Gate - Train the retrained candidate model using pinned hyperparameter configurations. - Compare candidate against the active production champion (`models:/fraud_detector/Production`) on the identical out-of-time validation dataset. - Candidate must pass all three gating criteria to qualify for promotion: 1. Primary Metric: PR-AUC improves by $\ge 0.015$ over the champion. 2. Guardrail Slice: High-value transaction recall ($\ge \$500$) must not drop by more than $0.5\%$. 3. Latency Budget: Inference latency p95 is $\le 30 ext{ms}$. ### Deliverable 4: Governed Registry Promotion & Webhook Notification - If the candidate passes all gates: - Register as new version in MLflow Model Registry under `fraud_detector`. - Tag version with `trigger_type`, `training_window`, `pr_auc_lift`, and `champion_version`. - Transition model version to `Staging`. - Emit an automated notification payload for human sign-off prior to production traffic cutover. --- ## Starter Code Scaffold ```python import argparse import mlflow import mlflow.sklearn from datetime import datetime, timedelta import pandas as pd from sklearn.metrics import precision_recall_curve, auc, recall_score def evaluate_gate(candidate_model, champion_model, val_df, feature_cols, target_col): X_val = val_df[feature_cols] y_val = val_df[target_col] # 1. PR-AUC Evaluation cand_probs = candidate_model.predict_proba(X_val)[:, 1] champ_probs = champion_model.predict_proba(X_val)[:, 1] p_c, r_c, _ = precision_recall_curve(y_val, cand_probs) p_ch, r_ch, _ = precision_recall_curve(y_val, champ_probs) cand_pr_auc = auc(r_c, p_c) champ_pr_auc = auc(r_ch, p_ch) pr_auc_passed = (cand_pr_auc - champ_pr_auc) >= 0.015 # 2. High-Value Slice Recall high_val_mask = val_df["amount"] >= 500.0 cand_slice_recall = recall_score(y_val[high_val_mask], cand_probs[high_val_mask] >= 0.5) champ_slice_recall = recall_score(y_val[high_val_mask], champ_probs[high_val_mask] >= 0.5) slice_passed = (champ_slice_recall - cand_slice_recall) <= 0.005 return { "cand_pr_auc": cand_pr_auc, "champ_pr_auc": champ_pr_auc, "pr_auc_lift": cand_pr_auc - champ_pr_auc, "passed_gate": pr_auc_passed and slice_passed } ``` --- ## 100-Point Graded Rubric * **Criteria 1: Pipeline Orchestration & Trigger Modality (25 pts)** - *25 pts:* Implements modular pipeline supporting both scheduled and drift-triggered execution; enforces sample volume assertions. - *15 pts:* Script executes but does not support multiple trigger modalities or volume checks. - *0 pts:* Script fails to execute or contains hardcoded static paths. * **Criteria 2: Temporal Window Partitioning (25 pts)** - *25 pts:* Strictly partitions rolling 180d training and 30d out-of-time validation windows; zero temporal leakage. - *15 pts:* Uses random train-test splitting rather than temporal out-of-time partitions. - *0 pts:* Demonstrates obvious data leakage between training and evaluation windows. * **Criteria 3: Multi-Objective Champion-Challenger Gate (25 pts)** - *25 pts:* Compares candidate directly against active production champion; asserts both aggregate PR-AUC lift and high-value slice recall protection. - *15 pts:* Evaluates candidate against static metric threshold without direct champion comparison. - *0 pts:* Auto-promotes candidate without validation gating. * **Criteria 4: Registry Governance & Notification (25 pts)** - *25 pts:* Stages approved models in MLflow Model Registry with complete lineage tags; outputs structured promotion notification. - *15 pts:* Regis