SUBMISSION: Capstone 2 — Baseline-to-Boosted Model Comparison & Validation Suite

🎯 Hands-on Portfolio Deliverable (Capstone). Senior ML judgment in one project: never ship the first model that "works" — build the ladder (dummy → linear → forest → boosted), measure everything that matters (ranking, calibration, cost), and justify the champion like your on-call depends on it.

🏢 Business Scenario & Executive Prompt

PaySecure processes 2M UPI transactions/day with 0.3% fraud. The current rules engine catches 40% of fraud at a 12:1 false-alarm ratio, and analysts drown. The VP Engineering will fund ML iff you prove, under one locked validation suite: a ranked model comparison, calibrated probabilities (block/allow thresholds move money), p99 inference latency under 80 ms, and a 90-day monitoring plan. No cherry-picked metrics.

📋 Source Data & Input Schemas

paysecure_txn.csv — 1,000,000 transactions over 60 days (mock shape; generate with the scaffold):

Expected shapes: fraud concentrates in new-device + night-hour + micro-amount slices; strong day-of-week seasonality (time axis matters).

🎯 Required Deliverables

1. validation_suite.ipynb — runnable: locked splits, all 5+ contenders, PR/ROC + calibration + latency tables and plots.
2. model_comparison.csv — one row per contender: model, roc_auc, pr_auc, brier, precision_at_k, recall_at_k, p99_latency_ms, train_cost_min.
3. champion.pkl — winning pipeline (preprocessing + model), with suite_config.json (seeds, splits, thresholds).
4. monitoring_plan.md — drift monitors (PSI/alert thresholds), label-delay handling, retrain triggers, rollback procedure.

🛠️ Step-by-Step Implementation Guide

1. Lock the suite first. Time-based splits (first 40 days train, next 10 valid, last 10 test) + stratified group awareness; operating point K from analyst capacity. Suite config committed before any model trains.
2. Climb the ladder. Dummy → logistic (balanced) → random forest → XGBoost → LightGBM. Same suite, same K, no per-model peeking at test.
3. Measure three axes. Ranking (PR-AUC primary, ROC-AUC secondary), calibration (Brier + reliability — thresholds move money), cost (p99 latency, train time). A champion must clear all three bars.
4. Tune only the finalists. Random/Bayesian search on the top-2 under the locked suite; report tuned-vs-untuned delta to prove the search earned its compute.
5. Crown with math. Expected-loss comparison vs the rules engine (fraud caught × value − false alarms × review cost). If ML doesn't beat rules by a margin, say so.
6. Plan the afterlife. PSI monitors per top feature, prediction-rate alerts, 90-day label-delay retraining cadence, champion/challenger rollback path.

💻 Complete Runnable Starter Scaffold

import time, joblib, pandas as pd
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier
from lightgbm import LGBMClassifier
from sklearn.metrics import average_precision_score, brier_score_loss

CONTENDERS = {
    "dummy": DummyClassifier(strategy="stratified"),
    "logreg_bal": LogisticRegression(class_weight="balanced", max_iter=2000),
    "rf": RandomForestClassifier(n_estimators=300, min_samples_leaf=50, n_jobs=-1, random_state=7),
    "xgb": XGBClassifier(n_estimators=600, learning_rate=0.05, max_depth=6,
                         scale_pos_weight=332, subsample=0.8, random_state=7),
    "lgbm": LGBMClassifier(n_estimators=600, learning_rate=0.05, num_leaves=63,
                           scale_pos_weight=332, subsample=0.8, random_state=7),
}
rows = []
for name, clf in CONTENDERS.items():
    t0 = time.time()
    clf.fit(Xtr, ytr)
    p = clf.predict_proba(Xva)[:, 1]
    t1 = time.time()
    rows.append({"model": name,
                 "pr_auc": round(float(average_precision_score(yva, p)), 4),
                 "brier": round(float(brier_score_loss(yva, p)), 5),
                 "train_s": round(t1 - t0, 1)})
comp = pd.DataFrame(rows).sort_values("pr_auc", ascending=False)
comp.to_csv("model_comparison.csv", index=False)
print(comp)

🧪 Acceptance Criteria & Verification Checks

• model_comparison.csv has ≥ 5 contenders scored under the locked suite (PR-AUC + Brier + latency/train-cost columns present).
• Suite uses time-based (or group-aware) splits; no contender touched test before crowning; tuning delta is reported, not assumed.
• Champion beats the rules-engine baseline on expected loss with the math shown; otherwise a documented no-ship recommendation.
• monitoring_plan.md defines PSI/alert thresholds, label-delay retraining, and a rollback path; champion.pkl reloads and scores one row.

📊 Evaluation Rubric (100 points — passing ≥ 80)

📤 Submission Guidelines

1. Upload your files (.ipynb, .py, .pkl, .csv, .md, .png) via the file dropzone on this lesson.
2. Paste your public GitHub repo URL in the submission notes (repo must contain the same files + a 5-line README with the run command).
3. Name files exactly as listed under Required Deliverables — review checks filenames first.
4. Passing score is 80/100. You may resubmit; the highest score counts.