There are five mistakes that show up in real product analytics again and again.
- Simpson's Paradox - aggregate trend reverses when broken down by segments.
- Base-Rate Fallacy - ignoring base rates when interpreting accuracy/precision.
- Regression to the Mean - extreme measurements naturally relax back to average.
- Multiple Testing - testing 20 cuts at α = 0.05 guarantees ~1 false positive by pure chance.
- Selection & Survivorship Bias - non-random sampling missing key churned or non-engaged populations.
Multiple testing & Simpson's paradox in Python
import pandas as pd
from statsmodels.stats.multitest import multipletests
# 1. Multiple Testing: 20 metric cuts evaluated at alpha = 0.05
raw_p_values = [0.002, 0.041, 0.048, 0.12, 0.25, 0.038, 0.54, 0.88, 0.049, 0.31,
0.19, 0.07, 0.44, 0.65, 0.032, 0.77, 0.91, 0.15, 0.22, 0.045]
# Bonferroni adjustment: alpha / N
rejected, p_corrected, _, _ = multipletests(raw_p_values, alpha=0.05, method="bonferroni")
print(f"Original cuts with p < 0.05: {sum(p < 0.05 for p in raw_p_values)} of 20")
print(f"Confirmed significant after Bonferroni correction: {sum(rejected)} of 20")
# 2. Simpson's Paradox Verification in Pandas
df = pd.DataFrame({
"channel": ["Mobile App", "Mobile App", "Desktop Web", "Desktop Web"],
"arm": ["Control", "Treatment", "Control", "Treatment"],
"conversions": [100, 110, 800, 820],
"visitors": [2000, 2000, 4000, 4000]
})
df["cvr"] = df["conversions"] / df["visitors"]
print("
Segment-level CVR:
", df.to_string(index=False))
Takeaway
Always run through the 5-trap mental checklist before presenting any surprising finding to leadership.