Reading Confidence Intervals

# Reading Confidence Intervals A p-value is a yes/no answer to one narrow question. A **confidence interval** is a range of plausible values for the thing you actually care about. It's the more useful number in 90% of DA situations. ## What a CI is A 95% confidence interval for a lift might read: **+4.2% [+1.8%, +6.6%]**. Read that as: "based on the data, the true lift is plausibly somewhere between +1.8% and +6.6%, with our best point estimate being +4.2%." ## Computing 95% Confidence Intervals in Python ```python import statsmodels.stats.proportion as smp # Control: 1,230 / 10,000 (12.30%) # Treatment: 1,310 / 10,000 (13.10%) count_t, nobs_t = 1310, 10000 count_c, nobs_c = 1230, 10000 # Compute 95% Confidence Interval for (p_treatment - p_control) ci_low, ci_high = smp.confint_proportions_2indep( count1=count_t, nobs1=nobs_t, count2=count_c, nobs2=nobs_c, method="wald", compare="diff", alpha=0.05 ) point_diff = (count_t / nobs_t) - (count_c / nobs_c) print("--- 95% Confidence Interval for Absolute Lift ---") print(f"Point Estimate: +{point_diff:.2%}") print(f"95% CI: [+{ci_low:.2%}, +{ci_high:.2%}]") if ci_low > 0: print("✅ Decision Rule: Lower bound > 0 -> Confirms positive impact with 95% confidence.") ``` ## Why CIs beat p-values for communication A CI tells your stakeholder: - The effect is real (the interval doesn't cross zero). - The effect is at least +1.8% and possibly up to +6.6%. - They can make a business decision based on the floor of the range. ## Takeaway For any result that comes from a sample: 1. The point estimate. 2. The 95% CI. 3. The p-value if requested, but lead with the CI.