Case 4 — Set Up Drift Detection on Production Predictions

Enterprise Scenario: A fintech lending platform's credit risk model has been serving live predictions for 90 days. Because loan default ground truth arrives 120 days post-issuance, standard classification accuracy metrics cannot provide early warning of model decay. Marketing recently expanded user acquisition campaigns to new demographic segments. You must engineer an automated drift monitoring pipeline: calculate baseline reference distributions from training data, run Kolmogorov-Smirnov and Chi-square hypothesis tests, compute Population Stability Index (PSI) across incoming prediction windows, and generate actionable alerts.

Technical Requirements & Deliverables

Deliverable 1: Baseline Profiler (build_baseline.py)

• Ingest historical training data to generate an immutable reference profile:
  • Continuous Features: 10-quantile bin edges, means, and standard deviations.
  • Categorical Features: Frequency proportions per category.
  • Predictions: Baseline output score quantiles (p10, p50, p90).
• Serialize reference profile to baseline_profile.json.

Deliverable 2: Statistical Drift Engine (drift_monitor.py)

• Implement two-sample Kolmogorov-Smirnov (KS) test for all continuous numerical features (scipy.stats.ks_2samp), flagging drift when $p < 0.05$.
• Implement Chi-square goodness-of-fit test for categorical features (scipy.stats.chisquare), flagging drift when $p < 0.05$.
• Implement Population Stability Index (PSI) using symmetric Jeffreys divergence with decile binning: $$ ext{PSI} = \sum_{i=1}^{k} (A_i - E_i) imes \ln\left(rac{A_i}{E_i} ight)$$ Include epsilon smoothing ($10^{-4}$) to prevent division by zero or $\ln(0)$.
• Categorize PSI values: $ ext{PSI} \le 0.10$ (No Drift), $0.10 < ext{PSI} \le 0.25$ (Moderate Drift), $ ext{PSI} > 0.25$ (Significant Drift).

Deliverable 3: Prediction Output & Proxy Telemetry

• Monitor output prediction score distributions (mean, p50, p90, high-risk flag percentage) across 7-day sliding windows.

Deliverable 4: Automated Audit Report & Alerting

• Generate a structured Markdown/JSON drift report highlighting drifted features, test statistics, and recommended engineering triage actions.

Starter Code Scaffold

import json
import numpy as np
import pandas as pd
from scipy import stats

def calculate_psi(baseline_arr: np.ndarray, target_arr: np.ndarray, num_bins: int = 10, epsilon: float = 1e-4) -> float:
    # Compute Population Stability Index (PSI) using baseline decile bin edges.
    quantiles = np.linspace(0, 100, num_bins + 1)
    bin_edges = np.percentile(baseline_arr, quantiles)
    bin_edges[0] -= 1e-5
    bin_edges[-1] += 1e-5
    
    # Calculate bucket counts
    base_counts, _ = np.histogram(baseline_arr, bins=bin_edges)
    targ_counts, _ = np.histogram(target_arr, bins=bin_edges)
    
    # Normalize to proportions with epsilon smoothing
    base_props = (base_counts / len(baseline_arr)) + epsilon
    targ_props = (targ_counts / len(target_arr)) + epsilon
    
    # Re-normalize
    base_props /= np.sum(base_props)
    targ_props /= np.sum(targ_props)
    
    # Compute symmetric PSI
    psi = np.sum((targ_props - base_props) * np.log(targ_props / base_props))
    return float(psi)

def audit_features(baseline_df: pd.DataFrame, target_df: pd.DataFrame, numerical_cols: list[str]) -> dict:
    results = {}
    for col in numerical_cols:
        ks_stat, ks_pval = stats.ks_2samp(baseline_df[col].dropna(), target_df[col].dropna())
        psi_val = calculate_psi(baseline_df[col].dropna().values, target_df[col].dropna().values)
        results[col] = {
            "ks_stat": float(ks_stat),
            "ks_pvalue": float(ks_pval),
            "ks_drift": bool(ks_pval < 0.05),
            "psi": float(psi_val),
            "psi_status": "Significant" if psi_val > 0.25 else "Moderate" if psi_val > 0.1 else "Stable"
        }
    return results

100-Point Graded Rubric

• Criteria 1: Baseline Profiling & Quantile Bins (25 pts)
  • 25 pts: Implements clean reference profiling with decile bin edges; serializes immutable reference metadata.
  • 15 pts: Computes basic means and variances but omits quantile bin edges for PSI.
  • 0 pts: Fails to isolate training baseline distribution.
• Criteria 2: Hypothesis Testing Implementation (25 pts)
  • 25 pts: Applies two-sample KS test on continuous features and Chi-square on categoricals; correctly interprets p-value thresholds.
  • 15 pts: Applies tests but misinterprets null hypothesis rejection criteria.
  • 0 pts: Implements incorrect statistical tests or hardcoded dummy checks.
• Criteria 3: Population Stability Index & Smoothing (25 pts)
  • 25 pts: Implements symmetric PSI formula with epsilon smoothing; correctly classifies drift into standard risk tiers.
  • 15 pts: Computes PSI but crashes on empty bins due to missing epsilon smoothing.
  • 0 pts: Formula mathematically incorrect.
• Criteria 4: Prediction Proxy Telemetry & Reporting (25 pts)
  • 25 pts: Tracks prediction output distributions across sliding windows; outputs actionable markdown triage report.
  • 15 pts: Evaluates feature drift but ignores output prediction distribution monitoring.
  • 0 pts: No reporting or alerting output generated.