SUBMISSION: Project 2 — End-to-End Churn Prediction with Cost-Sensitive Loss & ROC/PR Curves

🎯 Hands-on Portfolio Deliverable. The classic supervised-learning endgame: imbalanced labels, calibrated scores, and a threshold chosen by money — not 0.5. Your notebook answers "how do you handle class imbalance in production?"

🏢 Business Scenario & Executive Prompt

CallVeda, a telecom operator with 100K prepaid subscribers, loses 11% monthly. Retention can call 8,000 subscribers/month at ₹40/call; each saved subscriber is worth ₹900. The CMO asks: "Which 8,000 do we call, and what ROI will the program deliver?" A raw-accuracy model ("predict no-churn always: 89% accurate!") already fooled one vendor — yours must survive finance review.

📋 Source Data & Input Schemas

callveda_subs.csv — 100,000 rows (mock shape; generate with the scaffold):

Expected shapes: churners skew to low tenure + high support calls; region has 12 levels.

🎯 Required Deliverables

1. churn_model.ipynb — cleaned, re-runnable notebook: EDA, pipeline, ROC/PR, calibration, threshold economics.
2. churn_pipeline.pkl — fitted Pipeline (preprocessing + calibrated classifier), loadable with joblib.
3. metrics.json — roc_auc, pr_auc, brier, precision_at_8000, recall_at_8000, expected_roi_inr.
4. threshold_memo.md — one page: why 0.5 is wrong here, chosen threshold, ROI math finance can audit.

🛠️ Step-by-Step Implementation Guide

1. Split first. Stratified train/valid/test (60/20/20). All preprocessing inside a Pipeline — never fit on full data.
2. Baseline. Dummy (stratified) + plain logistic: record ROC-AUC and PR-AUC. Note how ROC flatters the dummy.
3. Go cost-sensitive. LogisticRegression(class_weight="balanced"); compare PR curves before/after. Optionally try sample_weight with the ₹900/₹40 cost ratio.
4. Calibrate. Wrap in CalibratedClassifierCV(cv=3); plot reliability curve + report Brier score — scores become callable money.
5. Price the threshold. Sweep thresholds on VALIDATION; compute precision@8000 and expected ROI; lock the threshold; report once on test.
6. Write the memo. Calls to make, expected saves, ROI range, and the two biggest risks (drift, offer fatigue).

💻 Complete Runnable Starter Scaffold

import numpy as np, pandas as pd, json, joblib
from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import roc_auc_score, average_precision_score, brier_score_loss

df = pd.read_csv("callveda_subs.csv")
X, y = df.drop(columns=["sub_id", "churned"]), df["churned"]
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, stratify=y, random_state=7)
Xtr, Xva, ytr, yva = train_test_split(Xtr, ytr, test_size=0.25, stratify=ytr, random_state=7)

num = ["tenure_months", "monthly_bill_inr", "data_gb_used", "support_calls_90d"]
cat = ["plan", "region"]
pre = ColumnTransformer([
    ("n", Pipeline([("imp", SimpleImputer(strategy="median")), ("sc", StandardScaler())]), num),
    ("c", Pipeline([("imp", SimpleImputer(strategy="most_frequent")),
                    ("oh", OneHotEncoder(handle_unknown="ignore"))]), cat),
])
base = Pipeline([("pre", pre),
                 ("clf", LogisticRegression(class_weight="balanced", max_iter=2000))])
model = CalibratedClassifierCV(base, cv=3).fit(Xtr, ytr)
p = model.predict_proba(Xte)[:, 1]
print("ROC-AUC", roc_auc_score(yte, p), "PR-AUC", average_precision_score(yte, p),
      "Brier", brier_score_loss(yte, p))
joblib.dump(model, "churn_pipeline.pkl")

🧪 Acceptance Criteria & Verification Checks

• metrics.json reports ROC-AUC, PR-AUC, Brier, precision/recall@8000, and ROI — test-set, threshold locked on validation.
• PR-AUC beats the dummy baseline by ≥ 5x prevalence (0.11); Brier improves after calibration.
• Chosen threshold ≠ 0.5 with cost math (₹40/call vs ₹900/save) shown in the memo.
• churn_pipeline.pkl loads in a fresh process and scores a 1-row DataFrame (unseen region level does not crash).

📊 Evaluation Rubric (100 points — passing ≥ 80)

📤 Submission Guidelines

1. Upload your files (.ipynb, .py, .pkl, .csv, .md, .png) via the file dropzone on this lesson.
2. Paste your public GitHub repo URL in the submission notes (repo must contain the same files + a 5-line README with the run command).
3. Name files exactly as listed under Required Deliverables — review checks filenames first.
4. Passing score is 80/100. You may resubmit; the highest score counts.