SUBMISSION: Capstone 2 — Baseline-to-Boosted Model Comparison & Validation Suite

# SUBMISSION: Capstone 2 — Baseline-to-Boosted Model Comparison & Validation Suite > 🎯 **Hands-on Portfolio Deliverable (Capstone).** Senior ML judgment in one project: never ship the first model that "works" — build the ladder (dummy → linear → forest → boosted), measure everything that matters (ranking, calibration, cost), and justify the champion like your on-call depends on it. ## 🏢 Business Scenario & Executive Prompt **PaySecure** processes 2M UPI transactions/day with 0.3% fraud. The current rules engine catches 40% of fraud at a 12:1 false-alarm ratio, and analysts drown. The VP Engineering will fund ML iff you prove, under one locked validation suite: a ranked model comparison, calibrated probabilities (block/allow thresholds move money), p99 inference latency under 80 ms, and a 90-day monitoring plan. No cherry-picked metrics. ## 📋 Source Data & Input Schemas **`paysecure_txn.csv`** — 1,000,000 transactions over 60 days (mock shape; generate with the scaffold): | column | dtype | description | |---|---|---| | `txn_id` | str | unique transaction | | `ts` | datetime | transaction timestamp | | `amount_inr` | float | transaction value | | `merchant_cat` | category | 24 categories | | `device_age_days` | int | device fingerprint age | | `hour` | int | hour of day (derived, allowed) | | `user_txn_count_7d` | int | ⚠️ point-in-time velocity (recompute per as-of) | | `is_fraud` | 0/1 | label: confirmed fraud (0.3% positive) | Expected shapes: fraud concentrates in new-device + night-hour + micro-amount slices; strong day-of-week seasonality (time axis matters). ## 🎯 Required Deliverables 1. **`validation_suite.ipynb`** — runnable: locked splits, all 5+ contenders, PR/ROC + calibration + latency tables and plots. 2. **`model_comparison.csv`** — one row per contender: `model, roc_auc, pr_auc, brier, precision_at_k, recall_at_k, p99_latency_ms, train_cost_min`. 3. **`champion.pkl`** — winning pipeline (preprocessing + model), with `suite_config.json` (seeds, splits, thresholds). 4. **`monitoring_plan.md`** — drift monitors (PSI/alert thresholds), label-delay handling, retrain triggers, rollback procedure. ## 🛠️ Step-by-Step Implementation Guide 1. **Lock the suite first.** Time-based splits (first 40 days train, next 10 valid, last 10 test) + stratified group awareness; operating point K from analyst capacity. Suite config committed before any model trains. 2. **Climb the ladder.** Dummy → logistic (balanced) → random forest → XGBoost → LightGBM. Same suite, same K, no per-model peeking at test. 3. **Measure three axes.** Ranking (PR-AUC primary, ROC-AUC secondary), calibration (Brier + reliability — thresholds move money), cost (p99 latency, train time). A champion must clear all three bars. 4. **Tune only the finalists.** Random/Bayesian search on the top-2 under the locked suite; report tuned-vs-untuned delta to prove the search earned its compute. 5. **Crown with math.** Expected-loss comparison vs the rules engine (fraud caught × value − false alarms × review cost). If ML doesn't beat rules by a margin, say so. 6. **Plan the afterlife.** PSI monitors per top feature, prediction-rate alerts, 90-day label-delay retraining cadence, champion/challenger rollback path. ## 💻 Complete Runnable Starter Scaffold ```python import time, joblib, pandas as pd from sklearn.dummy import DummyClassifier from sklearn.linear_model import LogisticRegression from sklearn.ensemble import RandomForestClassifier from xgboost import XGBClassifier from lightgbm import LGBMClassifier from sklearn.metrics import average_precision_score, brier_score_loss CONTENDERS = { "dummy": DummyClassifier(strategy="stratified"), "logreg_bal": LogisticRegression(class_weight="balanced", max_iter=2000), "rf": RandomForestClassifier(n_estimators=300, min_samples_leaf=50, n_jobs=-1, random_state=7), "xgb": XGBClassifier(n_estimators=600, learning_rate=0.05, max_depth=6, scale_pos_weight=332, subsample=0.8, random_state=7), "lgbm": LGBMClassifier(n_estimators=600, learning_rate=0.05, num_leaves=63, scale_pos_weight=332, subsample=0.8, random_state=7), } rows = [] for name, clf in CONTENDERS.items(): t0 = time.time() clf.fit(Xtr, ytr) p = clf.predict_proba(Xva)[:, 1] t1 = time.time() rows.append({"model": name, "pr_auc": round(float(average_precision_score(yva, p)), 4), "brier": round(float(brier_score_loss(yva, p)), 5), "train_s": round(t1 - t0, 1)}) comp = pd.DataFrame(rows).sort_values("pr_auc", ascending=False) comp.to_csv("model_comparison.csv", index=False) print(comp) ``` ## 🧪 Acceptance Criteria & Verification Checks - [ ] `model_comparison.csv` has ≥ 5 contenders scored under the locked suite (PR-AUC + Brier + latency/train-cost columns present). - [ ] Suite uses time-based (or group-aware) splits; no contender touched test before crowning; tuning delta is reported, not assume