Recap — Proxies and Indirect Metrics

# Proxies and Indirect Metrics Most production ML labels are delayed: - Churn prediction: confirms in weeks or months. - LTV forecast: confirms over the customer lifetime. - Fraud detection: confirms via chargeback in 30-90 days. - Click-through prediction on ads: same-day, but counting "conversion" takes longer. You can't measure accuracy directly until labels arrive. By then, the model may have been degrading for weeks. Proxies fill the gap. ## What proxies look like Three categories: ### 1. Output distribution proxies Stability of the prediction distribution itself. If the model usually predicts 5% positive at the median and today it's predicting 15%, something has changed. Track: - Mean prediction. - Median, p25, p75, p99 of predictions. - Histogram shape (compare bins to baseline). ```python def daily_prediction_stats(predictions_df): return { 'count': len(predictions_df), 'mean': predictions_df['prediction'].mean(), 'p50': predictions_df['prediction'].quantile(0.5), 'p95': predictions_df['prediction'].quantile(0.95), } # Compare to baseline baseline = { 'mean': 0.05, 'p50': 0.02, 'p95': 0.30, } if abs(daily_prediction_stats['mean'] - baseline['mean']) > 0.02: alert('Mean prediction drifted by > 2pp') ``` Sudden shifts in prediction distribution often correlate with bugs (feature pipeline) or drift. ### 2. Action-rate proxies What action does the system take based on predictions? Track that. For fraud detection: how many transactions are auto-blocked? For churn outreach: how many users does the outreach team contact? ```python daily_blocks = predictions_df.query('prediction > 0.95').shape[0] daily_blocks_per_1k = daily_blocks / (len(predictions_df) / 1000) ``` If today's block rate is 5x higher than baseline, something is off. Either real fraud spike, model drift, or a bug. Investigate. ### 3. Business-metric proxies Downstream KPIs that correlate with model performance: - Fraud model: chargebacks rate (but lagging). Customer support tickets about wrongly-blocked transactions (faster signal). - Churn outreach: completion rate of outreach (high uptake = relevant predictions). - Recommendation system: click-through rate, dwell time, conversion rate. These give earlier signals than the "true label" (chargeback / actual churn / etc.) and are often what the business actually cares about. ## Building a proxy dashboard Three rows of metrics for any ML system: ``` ┌──────────────────────────────────────────┐ │ Service health │ │ - Predictions/min, latency p99, errors │ ├──────────────────────────────────────────┤ │ Model behavior │ │ - Prediction mean, p50, p95 │ │ - Decision rate (% flagged) │ │ - Input feature PSI │ ├──────────────────────────────────────────┤ │ Business outcomes │ │ - Downstream KPIs │ │ - Customer support volume │ │ - True labels (lagging) │ └──────────────────────────────────────────┘ ``` Updated daily or hourly. Alerts on the top two rows fire fast; the bottom row confirms but slowly. ## When labels arrive Eventually labels come in. Backfill the truth: ```python # Week 1: log predictions predictions_2026_05_10 = log_to_warehouse(...) # Week 4: charge-backs arrive labels = load_labels(date='2026-05-10') # Compare historical = predictions_2026_05_10.merge(labels, on='transaction_id') historical_auc = roc_auc_score(historical['true_label'], historical['prediction']) if historical_auc < baseline_auc - 0.05: # Concept drift confirmed trigger_retraining() ``` Real performance verified, even with delay. ## Limitations of proxies Proxies aren't perfect indicators: - **A stable prediction distribution doesn't guarantee stable accuracy.** The model can output familiar-looking predictions for the wrong reasons. - **Business metrics have many drivers.** Conversion drops 10% — is it the model or the new pricing? - **Action-rate alerts fire on real changes too.** A genuine fraud spike looks like a drift signal. Use multiple proxies together. No single one is definitive. ## Triggering retraining from proxies A drift signal alone doesn't require immediate retraining. Decision rules: 1. **One proxy alerts**: investigate, don't retrain yet. 2. **Multiple proxies alert across days**: schedule retraining within a week. 3. **Sustained drift + reduced realized accuracy**: retrain ASAP. 4. **Sudden cliff**: probable pipeline bug, fix the upstream issue, don't retrain on bad data. The right reaction matches the signal's confidence. ## What to NOT do 1. **Wait for labels before noticing problems.** By then, the model has been wrong for weeks. 2. **Treat any alert as a 4am pager.** Most are false; investigate during business hours. 3. **Trust a single proxy.** Always cross-check. 4. **Build a 100-metric dashboard nobody reads.** 5-10 well-c