Calibration: PR-AUC/Brier + Threshold-to-Decision

# Calibration: PR-AUC/Brier + Threshold-to-Decision Target: pick a decision threshold from a cost matrix and report PR-AUC + Brier alongside it. WE W3-W4 framing; playbook: Applied_ML_Cheatsheet. ## 1. Report the right metric for imbalance ROC-AUC flatters on 95/5 splits because true negatives dominate. PR-AUC focuses on the rare class you care about; Brier scores the probability quality itself. Report all three, decide on PR-AUC + Brier. ## 2. Thresholds come from costs, not 0.5 A 0.5 default assumes symmetric costs. Build the 2x2 cost matrix (false-positive cost vs false-negative cost: wasted outreach vs lost customer) and pick the threshold minimizing expected cost on validation — then freeze it before holdout. ## 3. Calibrate before you threshold Boosted trees are often miscalibrated (overconfident). Platt/isotonic calibration on a held-out slice, verified with a reliability curve, then the frozen threshold maps to honest probabilities. ## Recap PR-AUC + Brier (not ROC alone), threshold from the cost matrix frozen pre-holdout, and calibration with a reliability curve before any decision memo.