Anatomy of an A/B Test Result

Focus on: Sample Size per arm, Primary Lift Point Estimate, 95% Confidence Interval, p-value, and Guardrail Status.

A/B test results evaluation in Python

import numpy as np
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep

# Sample: 14-day experiment results
# Control: 1,420 converted / 20,000 visitors (7.10%)
# Treatment: 1,580 converted / 20,000 visitors (7.90%)
n_c, conv_c = 20000, 1420
n_t, conv_t = 20000, 1580

p_c = conv_c / n_c
p_t = conv_t / n_t
abs_lift = p_t - p_c
rel_lift = abs_lift / p_c

z_stat, p_value = proportions_ztest([conv_t, conv_c], [n_t, n_c])
ci_low, ci_high = confint_proportions_2indep(conv_t, n_t, conv_c, n_c, alpha=0.05)

print(f"Control Rate:   {p_c:.2%}")
print(f"Treatment Rate: {p_t:.2%}")
print(f"Absolute Lift:  {abs_lift:+.2%} (Relative: {rel_lift:+.2%})")
print(f"95% CI on Lift: [{ci_low:+.2%}, {ci_high:+.2%}]")
print(f"P-value:        {p_value:.4f}")

mde = 0.005 # 0.5pp
if p_value < 0.05 and ci_low > 0 and abs_lift >= mde:
    print("🚀 VERDICT: SHIP IT. Clears statistical significance and MDE bar.")
else:
    print("🛑 VERDICT: DO NOT SHIP.")

Takeaway

Five numbers tell the real story. Ignore noisy daily fluctuation graphs.