SUBMISSION: Project 4 — Data Leakage Forensic Audit on a Temporal Credit Default Dataset

# SUBMISSION: Project 4 — Data Leakage Forensic Audit on a Temporal Credit Default Dataset > 🎯 **Hands-on Portfolio Deliverable.** *"Our model has 0.97 AUC — should we ship it?"* is a career-defining question. This audit proves you can answer it with evidence: planted leaks, honest rebuild, quantified optimism gap. ## 🏢 Business Scenario & Executive Prompt **CredTrust**, a digital lender, built a default-prediction model reporting AUC 0.97 on 3 years of loan-month rows. The CRO wants to 4x loan disbursal on its back. Your audit mandate, due Friday: reproduce the 0.97, identify every leak with file/line evidence, rebuild the evaluation honestly, and state the shippable number — plus guardrails so it never regresses. ## 📋 Source Data & Input Schemas **`credtrust_loans.csv`** — 900,000 loan-month rows, 2023-01 → 2025-12 (mock shape): | column | dtype | description | |---|---|---| | `loan_id` | str | loan account (each loan has up to 36 monthly rows) | | `month` | date | observation month | | `principal_outstanding_inr` | float | balance at month start | | `dpd` | int | days past due at month start | | `credit_score_at_booking` | int | bureau score at origination | | `collections_flag_next90` | 0/1 | ⚠️ SUSPECT: derived using future info | | `avg_dpd_full_history` | float | ⚠️ SUSPECT: aggregate over all months incl. future | | `defaulted_12m` | 0/1 | label: default within 12 months of `month` (6% positive) | Expected shapes: defaults cluster in 2024 rate-hike vintages; each `loan_id` spans many rows (entity leakage surface). ## 🎯 Required Deliverables 1. **`leakage_audit.md`** — per-leak finding: location, mechanism, evidence (metric delta when removed), severity. 2. **`fixed_pipeline.ipynb`** — runnable: reproduces 0.97 first, then the honest rebuild (point-in-time features, GroupKFold/time splits, pipeline-encapsulated preprocessing). 3. **`honest_metrics.json`** — leaky vs honest ROC-AUC, PR-AUC, and the optimism gap; per-vintage breakdown. 4. **`eval_guardrails.py`** — CI-checkable assertions (no future timestamps, no entity overlap, transformers fit on train only). ## 🛠️ Step-by-Step Implementation Guide 1. **Reproduce first.** Rebuild the 0.97 exactly (random row-wise split, all columns) — an audit that cannot reproduce the claim has no standing. 2. **Hunt the leaks.** Suspects: `collections_flag_next90` (future-derived), `avg_dpd_full_history` (full-window aggregate), scaler/imputer fit pre-split, random row split letting one `loan_id` span train+valid, shuffling across vintages. 3. **Quantify each.** Drop/fix one leak at a time; record the AUC delta. Rank findings by optimism contribution. 4. **Rebuild honestly.** Point-in-time features only; `GroupKFold` by `loan_id` (or time-based vintage splits); preprocessing inside `Pipeline`; stratified folds for the 6% label. 5. **Report the gap.** Leaky 0.97 vs honest number side by side, per vintage — plus the disbursal recommendation (ship / ship-with-limits / block). 6. **Lock the guardrails.** `eval_guardrails.py` asserts: feature max-ts ≤ label as-of, zero `loan_id` overlap, no pre-split fitting. Fails loud in CI. ## 💻 Complete Runnable Starter Scaffold ```python import pandas as pd from sklearn.model_selection import GroupKFold, cross_val_score from sklearn.pipeline import Pipeline from sklearn.ensemble import HistGradientBoostingClassifier df = pd.read_csv("credtrust_loans.csv", parse_dates=["month"]) # --- Guardrail 1: no entity overlap across folds --- # GroupKFold by loan_id — the SAME loan never trains and validates together. groups = df["loan_id"] X = df.drop(columns=["loan_id", "month", "defaulted_12m"]) y = df["defaulted_12m"] # --- Guardrail 2: drop future-derived columns (audit findings #1-2) --- SUSPECT = ["collections_flag_next90", "avg_dpd_full_history"] Xc = X.drop(columns=[c for c in SUSPECT if c in X.columns]) pipe = Pipeline([("clf", HistGradientBoostingClassifier(random_state=7))]) scores = cross_val_score(pipe, Xc, y, groups=groups, cv=GroupKFold(n_splits=5), scoring="roc_auc") print("honest ROC-AUC:", scores.mean().round(4), "+/-", scores.std().round(4)) ``` ## 🧪 Acceptance Criteria & Verification Checks - [ ] `leakage_audit.md` documents ≥ 3 leaks, each with mechanism + metric-delta evidence. - [ ] Honest rebuild uses point-in-time features, `GroupKFold`/`StratifiedGroupKFold` or vintage splits, pipeline-encapsulated preprocessing. - [ ] `honest_metrics.json` shows leaky vs honest ROC-AUC + PR-AUC with per-vintage splits; optimism gap is explained, not hidden. - [ ] `eval_guardrails.py` runs green on the honest pipeline and red on the leaky variant. ## 📊 Evaluation Rubric (100 points — passing ≥ 80) | # | Criterion | Points | What earns full marks | |---|---|---|---| | 1 | Modeling correctness | 25 | Right algorithm for the task; leakage-free splits/pipelines; metrics match the business decision; numbers reproducible | | 2 | Code qu