Case 6 — Run an A/B Test Between Two Model Versions

# Case 6 — Run an A/B Test Between Two Model Versions > *"You have model v1.2.0 in production and model v1.3.0 ready to test. Design and run a proper A/B test to determine whether v1.3.0 is genuinely better in production. Walk through setup, monitoring, and decision rules."* --- ## Step 1 — Plan the experiment Before any code: define what you're testing. - **Hypothesis**: v1.3.0 improves PR-AUC by at least 0.02 over v1.2.0 in production. - **Primary metric**: PR-AUC on labeled fraud data. - **Guardrail metrics**: precision at threshold, false-positive rate, latency. - **Decision rule**: if v1.3.0's PR-AUC is at least v1.2.0 + 0.01 with no guardrail regression, ramp up. Otherwise, keep v1.2.0. Document this. Pre-register before looking at any data. ## Step 2 — Power analysis How long does the test need to run? For a 0.02 PR-AUC difference on fraud at 0.3% positive rate: - ~50K labeled positives per arm for reliable detection. - With 0.3% positive rate, that's ~17M total transactions per arm. - At 1M transactions/day, that's 17 days per arm = 34 days total at 50/50 split. If the test would take too long, consider: - Larger split (e.g., 50/50 instead of 5/95) — but more risk if v2 is bad. - Shorter analysis window with confidence intervals. For non-fraud domains (more positive rate), shorter is fine. ## Step 3 — Implement the routing The service routes traffic based on hash: ```python # app/main.py import hashlib CHAMPION_VERSION = 'v1.2.0' CHALLENGER_VERSION = 'v1.3.0' CHALLENGER_PCT = 5 # start at 5% champion_model = mlflow.sklearn.load_model(f'models:/fraud_detector/{CHAMPION_VERSION}') challenger_model = mlflow.sklearn.load_model(f'models:/fraud_detector/{CHALLENGER_VERSION}') def assign_variant(transaction_id): h = int(hashlib.md5(transaction_id.encode()).hexdigest(), 16) return 'challenger' if (h % 100) < CHALLENGER_PCT else 'champion' @app.post('/predict') def predict(req): features = build_features(req.dict()) variant = assign_variant(req.transaction_id) if variant == 'challenger': prob = challenger_model.predict_proba([features])[0][1] version = CHALLENGER_VERSION else: prob = champion_model.predict_proba([features])[0][1] version = CHAMPION_VERSION background_tasks.add_task( log_prediction, transaction_id=req.transaction_id, prediction=prob, model_version=version, variant=variant, ) return {'fraud_probability': prob, 'model_version': version} ``` Hash-based ensures the same transaction is always routed to the same variant — important for retries. ## Step 4 — Log everything for analysis Each prediction row has: - `transaction_id` - `model_version` - `variant` (champion / challenger) - `prediction` - `features` (or feature_hash for storage) - `served_at` Later, after labels arrive: - `actual_label` - `label_arrived_at` Stream to the warehouse: ```sql CREATE TABLE production.ab_predictions ( transaction_id TEXT, served_at TIMESTAMP, variant TEXT, -- champion or challenger model_version TEXT, prediction NUMERIC, -- joined later actual_label INTEGER, label_arrived_at TIMESTAMP ); ``` ## Step 5 — Daily metric refresh ```sql -- daily_ab_metrics.sql WITH labeled AS ( SELECT * FROM production.ab_predictions p LEFT JOIN production.labels l ON l.transaction_id = p.transaction_id WHERE p.served_at >= '2026-05-15' AND (l.label_arrived_at IS NOT NULL OR p.served_at < NOW() - INTERVAL '30 days') ), per_variant AS ( SELECT variant, COUNT(*) AS n, SUM(actual_label) AS positives, -- Compute AUC, PR-AUC via UDF or external library FROM labeled GROUP BY variant ) SELECT * FROM per_variant; ``` Run via a daily job. Plot variant metrics over time. ## Step 6 — Monitor during the test ### Critical guardrails (alert immediately) - Service errors per variant. - Latency per variant. - Prediction distribution per variant (should look similar to expected). If challenger's error rate jumps or latency degrades — halt and investigate. ### Slow-confirmation metrics (review weekly) - PR-AUC per variant. - Precision at the operating threshold. - False-positive rate. - Business KPIs (fraud rate caught, chargeback rate). Confidence intervals on each: ```python from sklearn.utils import resample import numpy as np def bootstrap_ci(y_true, y_pred, n_boot=1000, alpha=0.05): scores = [] for _ in range(n_boot): idx = np.random.choice(len(y_true), len(y_true), replace=True) scores.append(roc_auc_score(y_true[idx], y_pred[idx])) return np.percentile(scores, [alpha/2*100, (1-alpha/2)*100]) champion_data = labeled.query('variant == "champion"') challenger_data = labeled.query('variant == "challenger"') champ_pr = bootstrap_ci(champion_data.actual_label, champion_data.prediction) chal_pr = bootstrap_ci(challenger_data.actual_label, challenger_data.prediction)