Reading P-values Without Getting Fooled

# Reading P-values Without Getting Fooled A new signup flow A/B test reports: *control 12.3%, treatment 13.1%, p = 0.04.* What does that p-value actually tell you? Almost certainly not what you think. ## The null hypothesis Every statistical test starts with a **null hypothesis** - the boring assumption that nothing is happening. For an A/B test, the null is "the two variants have the same true conversion rate". The data we collected gives us evidence to either reject the null or fail to reject it. The **alternative hypothesis** is the interesting one - "the treatment is actually different from control". ## What a p-value actually is A p-value is: *the probability of observing data this extreme or more extreme, assuming the null hypothesis is true.* If p = 0.04, what it says is: "if there were really no difference between control and treatment, we'd see a gap this big or bigger about 4% of the time just by chance." Small p-value means the observed gap is unlikely under the null. So we *reject* the null and act as though the treatment is different. ## Hypothesis testing in Python ```python import numpy as np from scipy import stats from statsmodels.stats.proportion import proportions_ztest # A/B Test: Control vs Treatment Signup Conversion # Control: 1,230 conversions out of 10,000 visitors (12.30%) # Treatment: 1,310 conversions out of 10,000 visitors (13.10%) count = np.array([1310, 1230]) nobs = np.array([10000, 10000]) # Two-sample z-test for proportions z_stat, p_value = proportions_ztest(count=count, nobs=nobs, alternative="two-sided") lift = (1310 / 10000) - (1230 / 10000) relative_lift = lift / (1230 / 10000) print(f"Observed Lift: +{lift:.2%} (Relative: +{relative_lift:.2%})") print(f"Z-statistic: {z_stat:.3f}") print(f"P-value: {p_value:.4f}") if p_value 0.05 means there's no effect."** No. It means you couldn't detect one with this sample. **3. "p < 0.05 means the effect is real."** No. It means the data is unlikely under the null. **4. "p is the probability of a false positive."** No. Your significance level α is the false-positive rate you'll accept over many tests. ## Takeaway P-values answer one specific, narrow question: how surprising is this data under the null? When in doubt, always report the confidence interval alongside.