SUBMISSION: Project 4 — Data Leakage Forensic Audit on a Temporal Credit Default Dataset

🎯 Hands-on Portfolio Deliverable. "Our model has 0.97 AUC — should we ship it?" is a career-defining question. This audit proves you can answer it with evidence: planted leaks, honest rebuild, quantified optimism gap.

🏢 Business Scenario & Executive Prompt

CredTrust, a digital lender, built a default-prediction model reporting AUC 0.97 on 3 years of loan-month rows. The CRO wants to 4x loan disbursal on its back. Your audit mandate, due Friday: reproduce the 0.97, identify every leak with file/line evidence, rebuild the evaluation honestly, and state the shippable number — plus guardrails so it never regresses.

📋 Source Data & Input Schemas

credtrust_loans.csv — 900,000 loan-month rows, 2023-01 → 2025-12 (mock shape):

Expected shapes: defaults cluster in 2024 rate-hike vintages; each loan_id spans many rows (entity leakage surface).

🎯 Required Deliverables

1. leakage_audit.md — per-leak finding: location, mechanism, evidence (metric delta when removed), severity.
2. fixed_pipeline.ipynb — runnable: reproduces 0.97 first, then the honest rebuild (point-in-time features, GroupKFold/time splits, pipeline-encapsulated preprocessing).
3. honest_metrics.json — leaky vs honest ROC-AUC, PR-AUC, and the optimism gap; per-vintage breakdown.
4. eval_guardrails.py — CI-checkable assertions (no future timestamps, no entity overlap, transformers fit on train only).

🛠️ Step-by-Step Implementation Guide

1. Reproduce first. Rebuild the 0.97 exactly (random row-wise split, all columns) — an audit that cannot reproduce the claim has no standing.
2. Hunt the leaks. Suspects: collections_flag_next90 (future-derived), avg_dpd_full_history (full-window aggregate), scaler/imputer fit pre-split, random row split letting one loan_id span train+valid, shuffling across vintages.
3. Quantify each. Drop/fix one leak at a time; record the AUC delta. Rank findings by optimism contribution.
4. Rebuild honestly. Point-in-time features only; GroupKFold by loan_id (or time-based vintage splits); preprocessing inside Pipeline; stratified folds for the 6% label.
5. Report the gap. Leaky 0.97 vs honest number side by side, per vintage — plus the disbursal recommendation (ship / ship-with-limits / block).
6. Lock the guardrails. eval_guardrails.py asserts: feature max-ts ≤ label as-of, zero loan_id overlap, no pre-split fitting. Fails loud in CI.

💻 Complete Runnable Starter Scaffold

import pandas as pd
from sklearn.model_selection import GroupKFold, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.ensemble import HistGradientBoostingClassifier

df = pd.read_csv("credtrust_loans.csv", parse_dates=["month"])

# --- Guardrail 1: no entity overlap across folds ---
# GroupKFold by loan_id — the SAME loan never trains and validates together.
groups = df["loan_id"]
X = df.drop(columns=["loan_id", "month", "defaulted_12m"])
y = df["defaulted_12m"]

# --- Guardrail 2: drop future-derived columns (audit findings #1-2) ---
SUSPECT = ["collections_flag_next90", "avg_dpd_full_history"]
Xc = X.drop(columns=[c for c in SUSPECT if c in X.columns])

pipe = Pipeline([("clf", HistGradientBoostingClassifier(random_state=7))])
scores = cross_val_score(pipe, Xc, y,
                         groups=groups,
                         cv=GroupKFold(n_splits=5),
                         scoring="roc_auc")
print("honest ROC-AUC:", scores.mean().round(4), "+/-", scores.std().round(4))

🧪 Acceptance Criteria & Verification Checks

• leakage_audit.md documents ≥ 3 leaks, each with mechanism + metric-delta evidence.
• Honest rebuild uses point-in-time features, GroupKFold/StratifiedGroupKFold or vintage splits, pipeline-encapsulated preprocessing.
• honest_metrics.json shows leaky vs honest ROC-AUC + PR-AUC with per-vintage splits; optimism gap is explained, not hidden.
• eval_guardrails.py runs green on the honest pipeline and red on the leaky variant.

📊 Evaluation Rubric (100 points — passing ≥ 80)

📤 Submission Guidelines

1. Upload your files (.ipynb, .py, .pkl, .csv, .md, .png) via the file dropzone on this lesson.
2. Paste your public GitHub repo URL in the submission notes (repo must contain the same files + a 5-line README with the run command).
3. Name files exactly as listed under Required Deliverables — review checks filenames first.
4. Passing score is 80/100. You may resubmit; the highest score counts.