- Peeking - stopping early inflates false positives by 3–5×.
- Novelty Effect - first-week spikes fade in week 2.
- Contamination - treatment leaks into control.
- Network Effects - interference between users across arms.
- Segment Surprises - overall lift masks segment regressions.
SRM & novelty checks in Python
from scipy import stats
import pandas as pd
# 1. Sample Ratio Mismatch (SRM) Chi-Square Test
# Target allocation: 50% / 50%. Observed: 24,100 Control vs 25,900 Treatment
n_ctrl_obs = 24100
n_treat_obs = 25900
total_users = n_ctrl_obs + n_treat_obs
expected_users = [total_users / 2, total_users / 2]
chi2_stat, srm_p_value = stats.chisquare(
f_obs=[n_ctrl_obs, n_treat_obs],
f_exp=expected_users
)
print("--- SRM Audit ---")
print(f"Observed: Control={n_ctrl_obs:,}, Treatment={n_treat_obs:,}")
print(f"Chi2 Stat: {chi2_stat:.3f}, SRM P-value: {srm_p_value:.6f}")
if srm_p_value < 0.01:
print("🚨 SRM DETECTED! Randomization broken. DO NOT trust test results until fixed.")
else:
print("✅ Sample allocation balanced. No SRM detected.")
# 2. Novelty Fade Check (Week 1 vs Week 2 Lift)
w1_lift = (720 / 10000) - (600 / 10000) # +1.20pp in Week 1
w2_lift = (610 / 10000) - (600 / 10000) # +0.10pp in Week 2 (Novelty fade!)
print(f"
Week 1 Lift: +{w1_lift:.2%} | Week 2 Lift: +{w2_lift:.2%}")
if w1_lift > 3 * w2_lift:
print("⚠️ Warning: Significant novelty fade detected between Week 1 and Week 2.")
Takeaway
Always run SRM and novelty checks before celebrating any experimental win.