The Five Traps

# The Five Stats Traps There are five mistakes that show up in real product analytics again and again. 1. **Simpson's Paradox** - aggregate trend reverses when broken down by segments. 2. **Base-Rate Fallacy** - ignoring base rates when interpreting accuracy/precision. 3. **Regression to the Mean** - extreme measurements naturally relax back to average. 4. **Multiple Testing** - testing 20 cuts at α = 0.05 guarantees ~1 false positive by pure chance. 5. **Selection & Survivorship Bias** - non-random sampling missing key churned or non-engaged populations. ## Multiple testing & Simpson's paradox in Python ```python import pandas as pd from statsmodels.stats.multitest import multipletests # 1. Multiple Testing: 20 metric cuts evaluated at alpha = 0.05 raw_p_values = [0.002, 0.041, 0.048, 0.12, 0.25, 0.038, 0.54, 0.88, 0.049, 0.31, 0.19, 0.07, 0.44, 0.65, 0.032, 0.77, 0.91, 0.15, 0.22, 0.045] # Bonferroni adjustment: alpha / N rejected, p_corrected, _, _ = multipletests(raw_p_values, alpha=0.05, method="bonferroni") print(f"Original cuts with p < 0.05: {sum(p < 0.05 for p in raw_p_values)} of 20") print(f"Confirmed significant after Bonferroni correction: {sum(rejected)} of 20") # 2. Simpson's Paradox Verification in Pandas df = pd.DataFrame({ "channel": ["Mobile App", "Mobile App", "Desktop Web", "Desktop Web"], "arm": ["Control", "Treatment", "Control", "Treatment"], "conversions": [100, 110, 800, 820], "visitors": [2000, 2000, 4000, 4000] }) df["cvr"] = df["conversions"] / df["visitors"] print(" Segment-level CVR: ", df.to_string(index=False)) ``` ## Takeaway Always run through the 5-trap mental checklist before presenting any surprising finding to leadership.