Case 4 — Diagnose a Model That Worked Yesterday but Not Today

# Case 4 — Diagnose a Model That Worked Yesterday but Not Today > *"Stakeholders are angry: 'The model is broken — predictions look wrong.' Yesterday it was fine. Walk through the diagnostic process and find the cause in under 30 minutes."* --- ## Step 1 — Confirm something is actually wrong Don't take "it looks wrong" at face value. Pull data: ```sql SELECT DATE(served_at) AS day, COUNT(*) AS predictions, AVG(churn_probability) AS mean_prob, PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY churn_probability) AS p50, PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY churn_probability) AS p95, COUNT(DISTINCT customer_id) AS unique_customers FROM production.predictions WHERE served_at >= NOW() - INTERVAL '7 days' GROUP BY 1 ORDER BY 1; ``` Result: ``` day predictions mean_prob p50 p95 unique_customers 2026-05-19 124,500 0.052 0.030 0.180 78,200 2026-05-20 127,300 0.051 0.028 0.175 79,500 2026-05-21 121,200 0.050 0.030 0.180 76,800 2026-05-22 125,800 0.049 0.029 0.175 78,900 2026-05-23 126,400 0.053 0.031 0.183 79,100 2026-05-24 124,000 0.052 0.030 0.180 78,400 2026-05-25 128,900 0.048 0.300 0.500 80,500 ← anomaly ``` Today's predictions look very different. Mean similar but p50 and p95 dramatically higher. Something changed. ## Step 2 — Check the obvious ### Did the model change? ```sql SELECT DISTINCT model_version FROM production.predictions WHERE served_at >= NOW() - INTERVAL '7 days'; ``` If today shows a new model_version, you've found it — a new deploy went out. Roll back, investigate. If model_version is unchanged: not a model issue. Look elsewhere. ### Did the service version change? ```bash gcloud run revisions list --service=churn-service ``` Recent deploys? Maybe environment differs. ### Did upstream data change? ```sql -- Check the source table volume SELECT DATE_TRUNC('day', updated_at) AS day, COUNT(*) FROM source.customers WHERE updated_at >= NOW() - INTERVAL '7 days' GROUP BY 1; ``` Big spike or drop? Schema change? New columns? ```sql -- Check the column count SELECT column_name FROM information_schema.columns WHERE table_name='customers'; ``` If columns have been added/removed/renamed, the feature pipeline is broken. ## Step 3 — Check feature distributions ```sql SELECT DATE(served_at) AS day, AVG(tenure_months) AS tenure, AVG(monthly_revenue) AS revenue, AVG(logins_last_30d) AS logins, AVG(feature_usage_score) AS usage, SUM(CASE WHEN feature_usage_score IS NULL THEN 1 ELSE 0 END) AS missing_usage FROM production.predictions WHERE served_at >= NOW() - INTERVAL '7 days' GROUP BY 1 ORDER BY 1; ``` If today shows: - `feature_usage` is suddenly all NULL → upstream pipeline broken. - `monthly_revenue` averages 4900 vs 49 → unit change (someone moved from dollars to cents). - `logins_last_30d` all zero → log pipeline broken. This is usually the diagnosis. A feature broke; the model is receiving garbage; predictions are wrong. ## Step 4 — Spot-check individual predictions ```sql SELECT customer_id, tenure_months, monthly_revenue, logins_last_30d, feature_usage_score, churn_probability FROM production.predictions WHERE served_at > NOW() - INTERVAL '1 hour' LIMIT 20; ``` Look at the values. Do they match what you'd expect for these customers? If predictions don't match the features (e.g., a customer with great tenure and low support tickets predicted 80% churn), investigate the model's recent inputs vs training inputs. ## Step 5 — Compare to training distribution ```python import pandas as pd training = pd.read_parquet('data/training_v3.parquet') today = load_recent_production(hours=24) for col in ['tenure_months', 'monthly_revenue', 'logins_last_30d', 'feature_usage_score']: print(f"{col}:") print(f" Training: mean={training[col].mean():.2f}, p50={training[col].quantile(0.5):.2f}") print(f" Today: mean={today[col].mean():.2f}, p50={today[col].quantile(0.5):.2f}") print(f" Ratio: {today[col].mean() / training[col].mean():.2f}x") ``` A 100× ratio on `monthly_revenue` is a unit conversion issue. A drop to zero is a pipeline failure. ## Step 6 — Find when it broke ```sql -- Hourly slice for the anomalous day SELECT DATE_TRUNC('hour', served_at) AS hour, AVG(feature_usage_score) AS usage, AVG(churn_probability) AS prob FROM production.predictions WHERE DATE(served_at) = '2026-05-25' GROUP BY 1 ORDER BY 1; ``` You see: usage averages 70 from 00:00-09:00, then drops to NULL from 10:00 onward. Predictions go off the rails at 10:00. Correlate with system events: - Was anything deployed at 10:00? - Was the feature pipeline updated? - Did a database migration run? ## Step 7 — Stop the bleeding While investigating, mitigate impact: ### Option A: Roll back to previous model If the deploy was the cause, this is the right action. ```bash gcloud run services update-traffic churn-service \