Anatomy of an A/B Test Result

# Anatomy of an A/B Test Result Focus on: Sample Size per arm, Primary Lift Point Estimate, 95% Confidence Interval, p-value, and Guardrail Status. ## A/B test results evaluation in Python ```python import numpy as np from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep # Sample: 14-day experiment results # Control: 1,420 converted / 20,000 visitors (7.10%) # Treatment: 1,580 converted / 20,000 visitors (7.90%) n_c, conv_c = 20000, 1420 n_t, conv_t = 20000, 1580 p_c = conv_c / n_c p_t = conv_t / n_t abs_lift = p_t - p_c rel_lift = abs_lift / p_c z_stat, p_value = proportions_ztest([conv_t, conv_c], [n_t, n_c]) ci_low, ci_high = confint_proportions_2indep(conv_t, n_t, conv_c, n_c, alpha=0.05) print(f"Control Rate: {p_c:.2%}") print(f"Treatment Rate: {p_t:.2%}") print(f"Absolute Lift: {abs_lift:+.2%} (Relative: {rel_lift:+.2%})") print(f"95% CI on Lift: [{ci_low:+.2%}, {ci_high:+.2%}]") print(f"P-value: {p_value:.4f}") mde = 0.005 # 0.5pp if p_value 0 and abs_lift >= mde: print("🚀 VERDICT: SHIP IT. Clears statistical significance and MDE bar.") else: print("🛑 VERDICT: DO NOT SHIP.") ``` ## Takeaway Five numbers tell the real story. Ignore noisy daily fluctuation graphs.