The Five Failure Modes

# The Five Failure Modes 1. **Peeking** - stopping early inflates false positives by 3–5×. 2. **Novelty Effect** - first-week spikes fade in week 2. 3. **Contamination** - treatment leaks into control. 4. **Network Effects** - interference between users across arms. 5. **Segment Surprises** - overall lift masks segment regressions. ## SRM & novelty checks in Python ```python from scipy import stats import pandas as pd # 1. Sample Ratio Mismatch (SRM) Chi-Square Test # Target allocation: 50% / 50%. Observed: 24,100 Control vs 25,900 Treatment n_ctrl_obs = 24100 n_treat_obs = 25900 total_users = n_ctrl_obs + n_treat_obs expected_users = [total_users / 2, total_users / 2] chi2_stat, srm_p_value = stats.chisquare( f_obs=[n_ctrl_obs, n_treat_obs], f_exp=expected_users ) print("--- SRM Audit ---") print(f"Observed: Control={n_ctrl_obs:,}, Treatment={n_treat_obs:,}") print(f"Chi2 Stat: {chi2_stat:.3f}, SRM P-value: {srm_p_value:.6f}") if srm_p_value < 0.01: print("🚨 SRM DETECTED! Randomization broken. DO NOT trust test results until fixed.") else: print("✅ Sample allocation balanced. No SRM detected.") # 2. Novelty Fade Check (Week 1 vs Week 2 Lift) w1_lift = (720 / 10000) - (600 / 10000) # +1.20pp in Week 1 w2_lift = (610 / 10000) - (600 / 10000) # +0.10pp in Week 2 (Novelty fade!) print(f" Week 1 Lift: +{w1_lift:.2%} | Week 2 Lift: +{w2_lift:.2%}") if w1_lift > 3 * w2_lift: print("⚠️ Warning: Significant novelty fade detected between Week 1 and Week 2.") ``` ## Takeaway Always run SRM and novelty checks before celebrating any experimental win.