Target: pick a decision threshold from a cost matrix and report PR-AUC + Brier alongside it. WE W3-W4 framing; playbook: Applied_ML_Cheatsheet.
1. Report the right metric for imbalance
ROC-AUC flatters on 95/5 splits because true negatives dominate. PR-AUC focuses on the rare class you care about; Brier scores the probability quality itself. Report all three, decide on PR-AUC + Brier.
2. Thresholds come from costs, not 0.5
A 0.5 default assumes symmetric costs. Build the 2x2 cost matrix (false-positive cost vs false-negative cost: wasted outreach vs lost customer) and pick the threshold minimizing expected cost on validation — then freeze it before holdout.
3. Calibrate before you threshold
Boosted trees are often miscalibrated (overconfident). Platt/isotonic calibration on a held-out slice, verified with a reliability curve, then the frozen threshold maps to honest probabilities.
Recap
PR-AUC + Brier (not ROC alone), threshold from the cost matrix frozen pre-holdout, and calibration with a reliability curve before any decision memo.