Case 4 — Set Up Drift Detection on Production Predictions

# Case 4 — Set Up Drift Detection on Production Predictions > **Enterprise Scenario:** A fintech lending platform's credit risk model has been serving live predictions for 90 days. Because loan default ground truth arrives 120 days post-issuance, standard classification accuracy metrics cannot provide early warning of model decay. Marketing recently expanded user acquisition campaigns to new demographic segments. You must engineer an automated drift monitoring pipeline: calculate baseline reference distributions from training data, run Kolmogorov-Smirnov and Chi-square hypothesis tests, compute Population Stability Index (PSI) across incoming prediction windows, and generate actionable alerts. --- ## Technical Requirements & Deliverables ### Deliverable 1: Baseline Profiler (`build_baseline.py`) - Ingest historical training data to generate an immutable reference profile: - Continuous Features: 10-quantile bin edges, means, and standard deviations. - Categorical Features: Frequency proportions per category. - Predictions: Baseline output score quantiles (p10, p50, p90). - Serialize reference profile to `baseline_profile.json`. ### Deliverable 2: Statistical Drift Engine (`drift_monitor.py`) - Implement two-sample Kolmogorov-Smirnov (KS) test for all continuous numerical features (`scipy.stats.ks_2samp`), flagging drift when $p < 0.05$. - Implement Chi-square goodness-of-fit test for categorical features (`scipy.stats.chisquare`), flagging drift when $p 0.25$ (Significant Drift). ### Deliverable 3: Prediction Output & Proxy Telemetry - Monitor output prediction score distributions (mean, p50, p90, high-risk flag percentage) across 7-day sliding windows. ### Deliverable 4: Automated Audit Report & Alerting - Generate a structured Markdown/JSON drift report highlighting drifted features, test statistics, and recommended engineering triage actions. --- ## Starter Code Scaffold ```python import json import numpy as np import pandas as pd from scipy import stats def calculate_psi(baseline_arr: np.ndarray, target_arr: np.ndarray, num_bins: int = 10, epsilon: float = 1e-4) -> float: # Compute Population Stability Index (PSI) using baseline decile bin edges. quantiles = np.linspace(0, 100, num_bins + 1) bin_edges = np.percentile(baseline_arr, quantiles) bin_edges[0] -= 1e-5 bin_edges[-1] += 1e-5 # Calculate bucket counts base_counts, _ = np.histogram(baseline_arr, bins=bin_edges) targ_counts, _ = np.histogram(target_arr, bins=bin_edges) # Normalize to proportions with epsilon smoothing base_props = (base_counts / len(baseline_arr)) + epsilon targ_props = (targ_counts / len(target_arr)) + epsilon # Re-normalize base_props /= np.sum(base_props) targ_props /= np.sum(targ_props) # Compute symmetric PSI psi = np.sum((targ_props - base_props) * np.log(targ_props / base_props)) return float(psi) def audit_features(baseline_df: pd.DataFrame, target_df: pd.DataFrame, numerical_cols: list[str]) -> dict: results = {} for col in numerical_cols: ks_stat, ks_pval = stats.ks_2samp(baseline_df[col].dropna(), target_df[col].dropna()) psi_val = calculate_psi(baseline_df[col].dropna().values, target_df[col].dropna().values) results[col] = { "ks_stat": float(ks_stat), "ks_pvalue": float(ks_pval), "ks_drift": bool(ks_pval 0.25 else "Moderate" if psi_val > 0.1 else "Stable" } return results ``` --- ## 100-Point Graded Rubric * **Criteria 1: Baseline Profiling & Quantile Bins (25 pts)** - *25 pts:* Implements clean reference profiling with decile bin edges; serializes immutable reference metadata. - *15 pts:* Computes basic means and variances but omits quantile bin edges for PSI. - *0 pts:* Fails to isolate training baseline distribution. * **Criteria 2: Hypothesis Testing Implementation (25 pts)** - *25 pts:* Applies two-sample KS test on continuous features and Chi-square on categoricals; correctly interprets p-value thresholds. - *15 pts:* Applies tests but misinterprets null hypothesis rejection criteria. - *0 pts:* Implements incorrect statistical tests or hardcoded dummy checks. * **Criteria 3: Population Stability Index & Smoothing (25 pts)** - *25 pts:* Implements symmetric PSI formula with epsilon smoothing; correctly classifies drift into standard risk tiers. - *15 pts:* Computes PSI but crashes on empty bins due to missing epsilon smoothing. - *0