Recap — Detecting and Acting on Drift

# Detecting and Acting on Drift Production models decay. Sometimes fast (a feature pipeline breaks), sometimes slow (market conditions shift). Two distinct kinds of decay: ## Data drift (covariate shift) The distribution of input features changes. Example: a fraud model trained on India transactions now serves US transactions; feature distributions shift. The relationship between features and target (P(y|X)) hasn't necessarily changed — but the inputs the model sees are different from what it trained on. **Symptom**: feature distribution monitoring shows shifts. Model accuracy may or may not degrade (depends on whether the model generalizes to the new feature region). **Causes**: - Pipeline upstream changed. - Customer base shifted. - Seasonality (holiday traffic, ad campaigns). - New product launches. ## Concept drift The relationship between features and target changes. Example: fraud patterns evolve — new attack methods, criminals adapt to detection. The features look the same, but the target's response to them is different. P(y|X) has changed. **Symptom**: accuracy degrades even though feature distributions look stable. **Causes**: - Adversarial adaptation (fraud, security). - Behavior changes (post-COVID consumer patterns). - Policy changes (new pricing, new product tiers). - Regulatory or market shifts. ## Why distinguishing matters Different problems, different fixes: - **Data drift**: retrain on the new distribution (the model is fine, the inputs are new). - **Concept drift**: retrain with newer labels (the model's mapping is stale). - **Both**: more comprehensive retraining. If you don't distinguish, you might retrain when nothing's actually broken (wasteful) or fail to retrain when you should (model decays further). ## Detecting data drift Compare current feature distribution to training distribution. ### Per-feature comparison For each feature: - **Kolmogorov-Smirnov test**: compares distributions, returns a p-value for "are they different?" - **Population Stability Index (PSI)**: a divergence measure. ```python def psi(expected, actual, buckets=10): expected_pct = pd.cut(expected, buckets).value_counts(normalize=True) actual_pct = pd.cut(actual, buckets).value_counts(normalize=True) # Avoid log(0) expected_pct = expected_pct.where(expected_pct > 0, 0.0001) actual_pct = actual_pct.where(actual_pct > 0, 0.0001) return ((actual_pct - expected_pct) * np.log(actual_pct / expected_pct)).sum() psi_score = psi(training_data['amount'], today_data['amount']) ``` PSI scale: - 0.25: significant drift, retrain. ### Multivariate detection Joint distributions can shift even if marginals look fine. Use: - **Maximum Mean Discrepancy (MMD)** or **kernel methods**: detect joint distribution differences. - **Adversarial classifier**: train a classifier to distinguish "training set" vs "today's data". If AUC > 0.6, there's drift. ```python adversarial_data = pd.concat([ training_data.assign(label=0), today_data.assign(label=1) ]) adversarial_model = XGBClassifier() score = cross_val_score(adversarial_model, X, y).mean() if score > 0.6: print('Significant distribution shift detected.') ``` ## Detecting concept drift (harder) Requires labels. Often labels arrive late. ### When labels are available Compute model performance on recent labeled data. Compare to baseline. ```python recent_predictions = predictions_table.query('date > now() - INTERVAL 7 days') recent_predictions = recent_predictions.merge(labels_table, on='id') recent_auc = roc_auc_score(recent_predictions['label'], recent_predictions['prediction']) baseline_auc = 0.85 # what we expect if recent_auc < baseline_auc - 0.05: print('Possible concept drift, AUC dropped from 0.85 to', recent_auc) ``` ### When labels are delayed You can't immediately verify accuracy. Proxy signals: - **Prediction distribution stability**: if model predictions look weird, performance likely is too. - **Confidence distribution**: model less confident than usual? Concept drift signal. - **Business metric drift**: if downstream KPIs change, the model may be a cause. For systems where labels arrive in weeks/months (churn, LTV), proxies are essential. ## What to monitor (the checklist) Daily / hourly checks: | Category | Metric | What it signals | |---|---|---| | Input | Feature distributions (PSI, mean, stddev) | Data drift | | Input | Missing value rate per feature | Pipeline failures | | Input | New categorical values | New patterns or data quality | | Output | Prediction distribution (mean, percentiles) | Major shift | | Output | Prediction count, latency | Service health | | Outcome | Action rate (e.g., % flagged) | Threshold drift | | Outcome | Realized accuracy on labeled data | Concept drift | | Business | KPI affected by model | Real-world impact | Tools: Evidently (open-source, Python-native), WhyLabs, Arize, custom dashboards on top of