A new signup flow A/B test reports: control 12.3%, treatment 13.1%, p = 0.04.
What does that p-value actually tell you? Almost certainly not what you think.
The null hypothesis
Every statistical test starts with a null hypothesis - the boring assumption that nothing is happening. For an A/B test, the null is "the two variants have the same true conversion rate". The data we collected gives us evidence to either reject the null or fail to reject it.
The alternative hypothesis is the interesting one - "the treatment is actually different from control".
What a p-value actually is
A p-value is: the probability of observing data this extreme or more extreme, assuming the null hypothesis is true.
If p = 0.04, what it says is: "if there were really no difference between control and treatment, we'd see a gap this big or bigger about 4% of the time just by chance." Small p-value means the observed gap is unlikely under the null. So we reject the null and act as though the treatment is different.
Hypothesis testing in Python
import numpy as np
from scipy import stats
from statsmodels.stats.proportion import proportions_ztest
# A/B Test: Control vs Treatment Signup Conversion
# Control: 1,230 conversions out of 10,000 visitors (12.30%)
# Treatment: 1,310 conversions out of 10,000 visitors (13.10%)
count = np.array([1310, 1230])
nobs = np.array([10000, 10000])
# Two-sample z-test for proportions
z_stat, p_value = proportions_ztest(count=count, nobs=nobs, alternative="two-sided")
lift = (1310 / 10000) - (1230 / 10000)
relative_lift = lift / (1230 / 10000)
print(f"Observed Lift: +{lift:.2%} (Relative: +{relative_lift:.2%})")
print(f"Z-statistic: {z_stat:.3f}")
print(f"P-value: {p_value:.4f}")
if p_value < 0.05:
print("✅ Statistically significant at α = 0.05 (Unlikely under the null hypothesis)")
else:
print("❌ Failed to reject the null hypothesis (Insufficient evidence)")
The four misinterpretations to avoid
1. "p = 0.04 means there's a 4% chance the null is true." No. P-values describe data given a hypothesis, not the probability of the hypothesis itself.
2. "p > 0.05 means there's no effect." No. It means you couldn't detect one with this sample.
3. "p < 0.05 means the effect is real." No. It means the data is unlikely under the null.
4. "p is the probability of a false positive." No. Your significance level α is the false-positive rate you'll accept over many tests.
Takeaway
P-values answer one specific, narrow question: how surprising is this data under the null? When in doubt, always report the confidence interval alongside.