Recap — Safely Rolling Out New Models

# Safely Rolling Out New Models Replacing a production model is risky. The new model might be: - Worse in production despite winning offline. - Wrong in some specific case the test set didn't cover. - Slower than the old one. - Less calibrated. Don't just swap. Roll out in stages. ## Stage 1: Shadow mode The new model (challenger) runs alongside the existing one (champion). The challenger's predictions are computed but NOT used for decisions. They're logged for comparison. ```python @app.post('/predict') def predict(req: TransactionRequest): features = build_features(req.dict()) # Champion (in use) champion_pred = champion_model.predict_proba([features])[0][1] # Challenger (shadow) challenger_pred = challenger_model.predict_proba([features])[0][1] # Log both for offline comparison log_prediction({ 'transaction_id': req.transaction_id, 'champion_pred': champion_pred, 'challenger_pred': challenger_pred, 'champion_version': CHAMPION_VERSION, 'challenger_version': CHALLENGER_VERSION, }) # Decision uses champion only return {'fraud_probability': float(champion_pred)} ``` What you check in shadow: - Are challenger predictions in a sensible range? - Are they correlated with champion predictions where they should be? - Where do they diverge? Are the divergences justified by features? - Does the challenger handle edge cases (high amounts, rare merchants) gracefully? Run shadow for ~1 week. Catches obvious bugs before any user is affected. ## Stage 2: Canary (small-percentage rollout) Once shadow looks good, route a small fraction of traffic to the challenger: ```python @app.post('/predict') def predict(req: TransactionRequest): features = build_features(req.dict()) # Decide which model based on hash of request ID use_challenger = (hash(req.transaction_id) % 100) < CANARY_PCT # e.g., 5% if use_challenger: prediction = challenger_model.predict_proba([features])[0][1] version = CHALLENGER_VERSION else: prediction = champion_model.predict_proba([features])[0][1] version = CHAMPION_VERSION log_prediction({ 'transaction_id': req.transaction_id, 'prediction': prediction, 'model_version': version, 'is_canary': use_challenger, }) return {'fraud_probability': float(prediction)} ``` CANARY_PCT starts at 1-5%. Monitor: - Business metrics for canary cohort vs champion cohort. - Latency. - Error rate. If anything's worse, halt the rollout. Investigate before increasing. ## Stage 3: Gradual ramp Increase canary_pct over days: - Day 1: 5% - Day 3: 10% - Day 5: 25% - Day 7: 50% - Day 10: 100% At each step, hold for at least a day to see business impact (which often lags behind prediction by hours-days). If anything regresses, roll back. ## What to monitor during rollout Three categories: ### Prediction-level - Distribution of predictions (should match historical baseline). - Confidence calibration if applicable. ### Action-level - Decision rate (how often is the model flagging fraud?). - Action mix (auto-block vs review vs ignore). ### Business-level - Real outcomes (when labeled): precision, recall. - KPIs (revenue lost to fraud, false-positive customer complaints). The business-level metrics often lag — fraud that happened today shows up as a chargeback in 30 days. You can't validate the new model's business impact in real-time. Monitor proxies (decision rate, distribution stability) and expect a slower confirmation cycle. ## What can go wrong ### 1. Champion-challenger correlation If both models share the same flawed feature, they both predict wrong together. Comparing them looks fine, but both are bad. Detect this with absolute monitoring (not just relative). ### 2. Trafficking the wrong way Hash-based assignment by transaction_id keeps the same transaction on the same model — needed for retries to be consistent. Random per-request would create flickering predictions. ### 3. Confounded comparisons The challenger gets seen primarily by Tuesday's traffic; champion by Monday's. Day-of-week effects bias the comparison. Mitigate by running long enough. ### 4. Calibration drift between models Champion outputs are calibrated at threshold 0.5; challenger outputs at 0.3. You can't directly compare. Either align thresholds or compare AUC / ranking quality. ## Holdout (control) groups for unbiased comparison A common refinement: even at 100% rollout, keep a 5% holdout on the old model permanently. Lets you measure the new model's incremental value over time. ```python @app.post('/predict') def predict(req: TransactionRequest): # Permanent holdout: 5% always on champion (old) is_holdout = (hash(req.user_id) % 100) < 5 use_new = not is_holdout if use_new: prediction = new_model.predict_proba(...) else: prediction = old_model.predict_proba(...) ``` After a month, compare metrics for new vs holdo