SUBMISSION: Project 2 — End-to-End Churn Prediction with Cost-Sensitive Loss & ROC/PR Curves

# SUBMISSION: Project 2 — End-to-End Churn Prediction with Cost-Sensitive Loss & ROC/PR Curves > 🎯 **Hands-on Portfolio Deliverable.** The classic supervised-learning endgame: imbalanced labels, calibrated scores, and a threshold chosen by money — not 0.5. Your notebook answers *"how do you handle class imbalance in production?"* ## 🏢 Business Scenario & Executive Prompt **CallVeda**, a telecom operator with 100K prepaid subscribers, loses 11% monthly. Retention can call 8,000 subscribers/month at ₹40/call; each saved subscriber is worth ₹900. The CMO asks: *"Which 8,000 do we call, and what ROI will the program deliver?"* A raw-accuracy model ("predict no-churn always: 89% accurate!") already fooled one vendor — yours must survive finance review. ## 📋 Source Data & Input Schemas **`callveda_subs.csv`** — 100,000 rows (mock shape; generate with the scaffold): | column | dtype | description | |---|---|---| | `sub_id` | str | unique subscriber | | `tenure_months` | int | months on network | | `monthly_bill_inr` | float | ARPU | | `data_gb_used` | float | last-month usage | | `support_calls_90d` | int | complaints in 90 days | | `plan` | category | `saver \| regular \| max` | | `region` | category | 12 circles (high-cardinality-ish) | | `churned` | 0/1 | label: ported out within 30 days (11% positive) | Expected shapes: churners skew to low tenure + high support calls; `region` has 12 levels. ## 🎯 Required Deliverables 1. **`churn_model.ipynb`** — cleaned, re-runnable notebook: EDA, pipeline, ROC/PR, calibration, threshold economics. 2. **`churn_pipeline.pkl`** — fitted `Pipeline` (preprocessing + calibrated classifier), loadable with `joblib`. 3. **`metrics.json`** — `roc_auc, pr_auc, brier, precision_at_8000, recall_at_8000, expected_roi_inr`. 4. **`threshold_memo.md`** — one page: why 0.5 is wrong here, chosen threshold, ROI math finance can audit. ## 🛠️ Step-by-Step Implementation Guide 1. **Split first.** Stratified train/valid/test (60/20/20). All preprocessing inside a `Pipeline` — never fit on full data. 2. **Baseline.** Dummy (stratified) + plain logistic: record ROC-AUC and PR-AUC. Note how ROC flatters the dummy. 3. **Go cost-sensitive.** `LogisticRegression(class_weight="balanced")`; compare PR curves before/after. Optionally try `sample_weight` with the ₹900/₹40 cost ratio. 4. **Calibrate.** Wrap in `CalibratedClassifierCV(cv=3)`; plot reliability curve + report Brier score — scores become callable money. 5. **Price the threshold.** Sweep thresholds on VALIDATION; compute precision@8000 and expected ROI; lock the threshold; report once on test. 6. **Write the memo.** Calls to make, expected saves, ROI range, and the two biggest risks (drift, offer fatigue). ## 💻 Complete Runnable Starter Scaffold ```python import numpy as np, pandas as pd, json, joblib from sklearn.model_selection import train_test_split from sklearn.compose import ColumnTransformer from sklearn.pipeline import Pipeline from sklearn.impute import SimpleImputer from sklearn.preprocessing import OneHotEncoder, StandardScaler from sklearn.linear_model import LogisticRegression from sklearn.calibration import CalibratedClassifierCV from sklearn.metrics import roc_auc_score, average_precision_score, brier_score_loss df = pd.read_csv("callveda_subs.csv") X, y = df.drop(columns=["sub_id", "churned"]), df["churned"] Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, stratify=y, random_state=7) Xtr, Xva, ytr, yva = train_test_split(Xtr, ytr, test_size=0.25, stratify=ytr, random_state=7) num = ["tenure_months", "monthly_bill_inr", "data_gb_used", "support_calls_90d"] cat = ["plan", "region"] pre = ColumnTransformer([ ("n", Pipeline([("imp", SimpleImputer(strategy="median")), ("sc", StandardScaler())]), num), ("c", Pipeline([("imp", SimpleImputer(strategy="most_frequent")), ("oh", OneHotEncoder(handle_unknown="ignore"))]), cat), ]) base = Pipeline([("pre", pre), ("clf", LogisticRegression(class_weight="balanced", max_iter=2000))]) model = CalibratedClassifierCV(base, cv=3).fit(Xtr, ytr) p = model.predict_proba(Xte)[:, 1] print("ROC-AUC", roc_auc_score(yte, p), "PR-AUC", average_precision_score(yte, p), "Brier", brier_score_loss(yte, p)) joblib.dump(model, "churn_pipeline.pkl") ``` ## 🧪 Acceptance Criteria & Verification Checks - [ ] `metrics.json` reports ROC-AUC, PR-AUC, Brier, precision/recall@8000, and ROI — test-set, threshold locked on validation. - [ ] PR-AUC beats the dummy baseline by ≥ 5x prevalence (0.11); Brier improves after calibration. - [ ] Chosen threshold ≠ 0.5 with cost math (₹40/call vs ₹900/save) shown in the memo. - [ ] `churn_pipeline.pkl` loads in a fresh process and scores a 1-row DataFrame (unseen `region` level does not crash). ## 📊 Evaluation Rubric (100 points — passing ≥ 80) | # | Criterion | Points | What earns full marks | |---|---|---|---| | 1 | Modeling correctness | 25 | Right algorithm for the task; leakage-free splits/pip