Statsig vs Shipsig

# Statsig vs Shipsig At Facebook scale, an A/B test on 10 million users can detect a +0.1% lift with p < 0.001. The result is statistically significant. The change is still not worth shipping. This is the gap between **statistical significance** (the math says there's an effect) and **practical significance** (the business cares about that effect). ## Evaluating Statsig vs Shipsig in Python ```python import statsmodels.stats.proportion as smp def evaluate_experiment(count_t, nobs_t, count_c, nobs_c, mde_absolute=0.015, alpha=0.05): # Evaluates whether an A/B test clears both Statsig AND Shipsig (MDE) p_t = count_t / nobs_t p_c = count_c / nobs_c lift = p_t - p_c z_stat, p_val = smp.proportions_ztest([count_t, count_c], [nobs_t, nobs_c]) ci_low, ci_high = smp.confint_proportions_2indep(count_t, nobs_t, count_c, nobs_c, alpha=alpha) is_statsig = (p_val 0) is_shipsig = lift >= mde_absolute print(f"Lift: {lift:+.2%} (95% CI: [{ci_low:+.2%}, {ci_high:+.2%}], p={p_val:.4f})") print(f"Statistically Significant (p < {alpha}): {is_statsig}") print(f"Practically Significant (Lift ≥ MDE {mde_absolute:.2%}): {is_shipsig}") if is_statsig and is_shipsig: return "🚀 SHIP: Meets statistical confidence and clears business MDE." elif is_statsig and not is_shipsig: return "🛑 DO NOT SHIP: Statistically significant but effect is below business MDE." else: return "⏳ INCONCLUSIVE: Not statistically significant. Do not ship." # Example: High traffic (100K users), tiny lift (+0.3pp), MDE target = +1.5pp print(evaluate_experiment(count_t=12300, nobs_t=100000, count_c=12000, nobs_c=100000, mde_absolute=0.015)) ``` ## Takeaway Statistical significance answers a math question. Practical significance answers the business question. Pre-specify MDE to prevent post-hoc rationalizations.