SUBMISSION: Capstone 1 — Model Interpretability & Bias Audit with SHAP Values

# SUBMISSION: Capstone 1 — Model Interpretability & Bias Audit with SHAP Values > 🎯 **Hands-on Portfolio Deliverable (Capstone).** Accuracy gets the demo; explainability gets the signature. This capstone proves you can open a black-box booster, defend individual decisions, and catch proxy discrimination before a regulator does. ## 🏢 Business Scenario & Executive Prompt **SahayCredit** approves 60K micro-loans/month with an XGBoost scorer (AUC 0.83). Internal review flags a pattern: approval rates for applicants from Tier-3 cities run 14 points below Tier-1 at equal credit scores. The risk head asks: *"Is the model biased, which features drive it, and what do we change — with proof the fix doesn't crater portfolio quality?"* Your audit lands on the risk committee's desk. ## 📋 Source Data & Input Schemas **`sahaycredit_apps.csv`** — 240,000 applications (mock shape; generate with the scaffold): | column | dtype | description | |---|---|---| | `app_id` | str | unique application | | `credit_score` | int | bureau score 300–900 | | `monthly_income_inr` | float | verified income | | `existing_emi_inr` | float | current obligations | | `employment_years` | float | tenure (5% missing) | | `city_tier` | category | `T1 \| T2 \| T3` | | `purpose` | category | `business \| education \| medical \| personal` | | `approved_repaid` | 0/1 | label: approved AND repaid in full (22% positive) | Expected shapes: Tier-3 applicants carry lower scores on average (confounding!); `purpose=medical` defaults more often. ## 🎯 Required Deliverables 1. **`shap_audit.ipynb`** — runnable: tuned XGBoost, global + local SHAP, segment-sliced importance, mitigation A/B. 2. **`best_model.pkl`** — final approved scorer (post-mitigation) with `model_features.json`. 3. **`fairness_table.csv`** — per-`city_tier`: approval rate, TPR, FPR, mean SHAP of top-5 features. 4. **`model_card.md`** — intended use, metrics + calibration, explainability summary, bias findings, mitigation, residual risks, monitoring plan. ## 🛠️ Step-by-Step Implementation Guide 1. **Baseline honestly.** Stratified split; XGBoost with early stopping on validation; report ROC-AUC, PR-AUC, Brier, and calibration — before any fairness talk. 2. **Explain globally.** `shap.TreeExplainer`: summary + dependence plots for the top features. Separate predictive signal from proxy signal. 3. **Explain locally.** Waterfall plots for 3 approved + 3 denied edge cases (near-threshold). Write the one-paragraph reason code for each, as a regulator would demand. 4. **Slice by segment.** Recompute importance and error rates per `city_tier`. Decompose the 14-point gap: how much is score-mix (confounding) vs model behavior? A feature whose SHAP pushes one segment down is a finding. 5. **Mitigate and re-measure.** Candidate fixes: drop/quarantine the proxy, reweigh, threshold per segment with equalized-odds check. Ship the variant with the best fairness-quality frontier — document what you traded. 6. **Write the model card.** Decision, numbers, residual risks (what you did NOT fix), and drift monitors (approval-rate gap alert at ±5 points). ## 💻 Complete Runnable Starter Scaffold ```python import pandas as pd, shap, joblib from sklearn.model_selection import train_test_split from xgboost import XGBClassifier df = pd.read_csv("sahaycredit_apps.csv") X = pd.get_dummies(df.drop(columns=["app_id", "approved_repaid"]), dtype=float) y = df["approved_repaid"] Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, stratify=y, random_state=7) clf = XGBClassifier(n_estimators=500, learning_rate=0.05, max_depth=5, subsample=0.8, colsample_bytree=0.8, scale_pos_weight=(ytr == 0).sum() / (ytr == 1).sum(), early_stopping_rounds=50, eval_metric="logloss", random_state=7) clf.fit(Xtr, ytr, eval_set=[(Xte, yte)], verbose=False) ex = shap.TreeExplainer(clf) sv = ex(Xte) # per-row, per-feature attributions # Global: shap.plots.beeswarm(sv, max_display=12) # Local: shap.plots.waterfall(sv[row_idx], max_display=10) # Sliced: sv[df.loc[Xte.index, "city_tier"] == "T3"].abs.mean(0) vs T1 joblib.dump(clf, "best_model.pkl") ``` ## 🧪 Acceptance Criteria & Verification Checks - [ ] Notebook shows global SHAP (top features + dependence) and ≥ 6 local waterfall reason codes. - [ ] `fairness_table.csv` quantifies the Tier-3 gap with approval rate, TPR/FPR, and sliced SHAP — confounding separated from model effect. - [ ] A mitigation variant is measured on both quality (PR-AUC/Brier within agreed tolerance) and fairness (gap narrowed); trade-off is explicit. - [ ] `model_card.md` covers use, metrics, findings, mitigation, residual risks, and monitoring thresholds. ## 📊 Evaluation Rubric (100 points — passing ≥ 80) | # | Criterion | Points | What earns full marks | |---|---|---|---| | 1 | Modeling correctness | 25 | Right algorithm for the task; leakage-free splits/pipelines; metrics match the business decision; numbers reproducible | | 2 |