Statsig vs Shipsig

At Facebook scale, an A/B test on 10 million users can detect a +0.1% lift with p < 0.001. The result is statistically significant. The change is still not worth shipping.

This is the gap between statistical significance (the math says there's an effect) and practical significance (the business cares about that effect).

Evaluating Statsig vs Shipsig in Python

import statsmodels.stats.proportion as smp

def evaluate_experiment(count_t, nobs_t, count_c, nobs_c, mde_absolute=0.015, alpha=0.05):
    # Evaluates whether an A/B test clears both Statsig AND Shipsig (MDE)
    p_t = count_t / nobs_t
    p_c = count_c / nobs_c
    lift = p_t - p_c

    z_stat, p_val = smp.proportions_ztest([count_t, count_c], [nobs_t, nobs_c])
    ci_low, ci_high = smp.confint_proportions_2indep(count_t, nobs_t, count_c, nobs_c, alpha=alpha)

    is_statsig = (p_val < alpha) and (ci_low > 0)
    is_shipsig = lift >= mde_absolute

    print(f"Lift: {lift:+.2%} (95% CI: [{ci_low:+.2%}, {ci_high:+.2%}], p={p_val:.4f})")
    print(f"Statistically Significant (p < {alpha}): {is_statsig}")
    print(f"Practically Significant (Lift ≥ MDE {mde_absolute:.2%}): {is_shipsig}")

    if is_statsig and is_shipsig:
        return "🚀 SHIP: Meets statistical confidence and clears business MDE."
    elif is_statsig and not is_shipsig:
        return "🛑 DO NOT SHIP: Statistically significant but effect is below business MDE."
    else:
        return "⏳ INCONCLUSIVE: Not statistically significant. Do not ship."

# Example: High traffic (100K users), tiny lift (+0.3pp), MDE target = +1.5pp
print(evaluate_experiment(count_t=12300, nobs_t=100000, count_c=12000, nobs_c=100000, mde_absolute=0.015))

Takeaway

Statistical significance answers a math question. Practical significance answers the business question. Pre-specify MDE to prevent post-hoc rationalizations.