A p-value is a yes/no answer to one narrow question. A confidence interval is a range of plausible values for the thing you actually care about. It's the more useful number in 90% of DA situations.
What a CI is
A 95% confidence interval for a lift might read: +4.2% [+1.8%, +6.6%].
Read that as: "based on the data, the true lift is plausibly somewhere between +1.8% and +6.6%, with our best point estimate being +4.2%."
Computing 95% Confidence Intervals in Python
import statsmodels.stats.proportion as smp
# Control: 1,230 / 10,000 (12.30%)
# Treatment: 1,310 / 10,000 (13.10%)
count_t, nobs_t = 1310, 10000
count_c, nobs_c = 1230, 10000
# Compute 95% Confidence Interval for (p_treatment - p_control)
ci_low, ci_high = smp.confint_proportions_2indep(
count1=count_t, nobs1=nobs_t,
count2=count_c, nobs2=nobs_c,
method="wald",
compare="diff",
alpha=0.05
)
point_diff = (count_t / nobs_t) - (count_c / nobs_c)
print("--- 95% Confidence Interval for Absolute Lift ---")
print(f"Point Estimate: +{point_diff:.2%}")
print(f"95% CI: [+{ci_low:.2%}, +{ci_high:.2%}]")
if ci_low > 0:
print("✅ Decision Rule: Lower bound > 0 -> Confirms positive impact with 95% confidence.")
Why CIs beat p-values for communication
A CI tells your stakeholder:
- The effect is real (the interval doesn't cross zero).
- The effect is at least +1.8% and possibly up to +6.6%.
- They can make a business decision based on the floor of the range.
Takeaway
For any result that comes from a sample:
- The point estimate.
- The 95% CI.
- The p-value if requested, but lead with the CI.