Topfolio
Tutorial

A/B Testing in Python: Sample Size, Hypothesis Testing & P-Value Analysis

Complete A/B testing guide for data analysts in Python. Calculate sample sizes with statsmodels, detect Sample Ratio Mismatch (SRM), run proportion z-tests and Welch's t-tests, and avoid the peeking problem.

Anuj SainiAug 24, 202614 min read

Tech giants like Netflix, Amazon, Booking.com, and Uber run tens of thousands of online controlled experiments every year. Why? Because product intuition fails more than 70% of the time.

Yet many data analysts make critical statistical errors: stopping tests early ("peeking"), skipping sample size estimation, misinterpreting p-values, or ignoring Sample Ratio Mismatch (SRM).

In this playbook, we'll walk through the end-to-end Python experimentation pipeline: from pre-experiment power calculations to hypothesis testing, confidence interval estimation, and post-experiment business decisions.

For broader foundations in Python analytics, check our Python Pandas Data Analysis Guide or enroll in the Data Analyst Career Track.

Experimentation on Topfolio Practice

Ready to test your statistical inference skills? Solve interactive Python hypothesis testing challenges on Topfolio Practice.

1. Experiment Foundations: Hypotheses, Alpha, and Power

Every controlled experiment balances two types of statistical errors:

                           REALITY IN POPULATION
                     H0 True (No Lift)      H1 True (Real Lift)
DECISION  Reject H0      TYPE I ERROR (α)        CORRECT DECISION
                       (False Positive)          (Power = 1 - β)
          ------------------------------------------------------
          Keep H0        CORRECT DECISION        TYPE II ERROR (β)
                         (True Negative)         (False Negative)
  • Null Hypothesis (H0): The variant has no effect on user behavior (p_variant = p_control or mu_variant = mu_control).
  • Alternative Hypothesis (H1): The variant produces a measurable difference (p_variant != p_control).
  • Significance Level ($\alpha = 0.05$): The probability of committing a Type I error (declaring a winning variant when none exists).
  • Statistical Power ($1 - \beta = 0.80$): The probability of correctly rejecting the null hypothesis when an effect actually exists.
  • Minimum Detectable Effect (MDE): The smallest relative percentage lift that matters to the business (e.g., +5% relative lift in checkout conversion).

2. Pre-Experiment Power Analysis: Sample Size Calculation

Never launch an A/B test without knowing exactly how many users each variant must collect. Testing without power calculation leads to either underpowered tests (missing real wins) or wasteful tests (burning weeks of traffic).

2.1 Sample Size for Binary Proportions (Conversion Rate / CTR)

Suppose our current checkout conversion rate is 10% ($p_1 = 0.10$), and product management wants to detect a relative lift of 20% (i.e., new rate $p_2 = 0.12$, an absolute lift of 2%).

python
import numpy as np
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize
 
# Parameters
baseline_cr = 0.10          # Current conversion rate: 10%
target_cr = 0.12            # Target conversion rate: 12% (+20% relative lift)
alpha = 0.05                # 5% significance level
power = 0.80                # 80% statistical power
 
# 1. Calculate Cohen's h effect size for proportions
effect_size = proportion_effectsize(prop1=target_cr, prop2=baseline_cr)
 
# 2. Compute sample size per variant
analysis = NormalIndPower()
sample_size_per_group = analysis.solve_power(
    effect_size=effect_size,
    alpha=alpha,
    power=power,
    ratio=1.0,               # 1:1 allocation (50% Control, 50% Variant)
    alternative='two-sided'
)
 
sample_size_per_group = int(np.ceil(sample_size_per_group))
total_sample_size = sample_size_per_group * 2
 
print("=" * 50)
print(f"BASELINE CONVERSION:       {baseline_cr * 100:.1f}%")
print(f"TARGET CONVERSION:         {target_cr * 100:.1f}% (MDE: +{((target_cr - baseline_cr)/baseline_cr)*100:.1f}%)")
print(f"REQUIRED SAMPLES / GROUP:  {sample_size_per_group:,} users")
print(f"TOTAL SAMPLE SIZE:         {total_sample_size:,} users")
print("=" * 50)
BASELINE CONVERSION:       10.0%
TARGET CONVERSION:         12.0% (MDE: +20.0%)
REQUIRED SAMPLES / GROUP:  3,835 users
TOTAL SAMPLE SIZE:         7,670 users

2.2 Sample Size for Continuous Metrics (Revenue / ARPU)

When testing continuous metrics (Average Revenue Per User), use Cohen's d = (mu_2 - mu_1) / sigma:

python
from statsmodels.stats.power import TTestIndPower
 
control_mean = 45.0         # Baseline ARPU = $45
variant_mean = 48.0         # Target ARPU = $48 (+$3 lift)
pooled_std = 25.0           # Standard deviation of spend
 
# Compute Cohen's d effect size
cohens_d = (variant_mean - control_mean) / pooled_std
 
ttest_power = TTestIndPower()
samples_arpu = ttest_power.solve_power(
    effect_size=cohens_d,
    alpha=0.05,
    power=0.80,
    ratio=1.0
)
 
print(f"Required Sample Size per Group for ARPU Test: {int(np.ceil(samples_arpu)):,} users")

3. Pre-Test Quality Check: Sample Ratio Mismatch (SRM)

Before analyzing conversion rates, you must verify that your randomization engine delivered traffic in the intended proportions.

If you designed a 50/50 test and observed 10,450 Control vs. 9,550 Variant users out of 20,000 total, is that natural variance or a broken pipeline?

3.1 Detecting SRM via Chi-Square Goodness of Fit

python
from scipy import stats
 
def check_srm(observed_control: int, observed_variant: int, expected_ratio=(0.5, 0.5)):
    total = observed_control + observed_variant
    expected_control = total * expected_ratio[0]
    expected_variant = total * expected_ratio[1]
    
    observed = [observed_control, observed_variant]
    expected = [expected_control, expected_variant]
    
    chi2_stat, p_val = stats.chisquare(f_obs=observed, f_exp=expected)
    
    print(f"Observed: Control={observed_control:,} | Variant={observed_variant:,}")
    print(f"Expected: Control={expected_control:,.0f} | Variant={expected_variant:,.0f}")
    print(f"SRM Test Chi2 Statistic: {chi2_stat:.4f}, p-value: {p_val:.6f}")
    
    # Industry convention uses p < 0.001 to flag SRM
    if p_val < 0.001:
        print("🚨 CRITICAL WARNING: Sample Ratio Mismatch (SRM) Detected!")
        print("Do NOT analyze this experiment. Investigate tracking or bot-filtering bugs.")
    else:
        print("✅ Randomization Integrity Validated. No SRM detected.")
 
# Example: 10,450 Control vs 9,550 Variant
check_srm(10450, 9550)
Observed: Control=10,450 | Variant=9,550
Expected: Control=10,000 | Variant=10,000
SRM Test Chi2 Statistic: 40.5000, p-value: 0.000000
🚨 CRITICAL WARNING: Sample Ratio Mismatch (SRM) Detected!

4. Scenario A: Conversion Rate Test (Discrete Binary Data)

Let's simulate a real-world checkout button test:

  • Control Group (Blue Button): 10.0% conversion rate ($n = 4,000$)
  • Variant Group (Red Button): 12.0% conversion rate ($n = 4,000$)
python
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
 
np.random.seed(42)
n_per_group = 4000
 
# Generate binomial outcomes (0 = abandoned, 1 = purchased)
control_conversions = np.random.binomial(n=1, p=0.10, size=n_per_group)
variant_conversions = np.random.binomial(n=1, p=0.12, size=n_per_group)
 
df_ab = pd.DataFrame({
    'Group': ['Control'] * n_per_group + ['Variant'] * n_per_group,
    'Converted': np.concatenate([control_conversions, variant_conversions])
})
 
# Aggregating metrics
summary = df_ab.groupby('Group').agg(
    Visitors=('Converted', 'count'),
    Conversions=('Converted', 'sum'),
    Conversion_Rate=('Converted', 'mean')
)
summary['Conversion_Rate_Pct'] = summary['Conversion_Rate'] * 100
print(summary)
GroupVisitorsConversionsConversion Rate (%)
Control4,0003829.55%
Variant4,00048712.18%

4.1 Visualizing Conversion Lift with 95% Confidence Intervals

python
plt.figure(figsize=(7, 5))
sns.barplot(
    data=df_ab,
    x='Group',
    y='Converted',
    ci=95,
    palette=['#3498db', '#e74c3c'],
    capsize=0.1
)
plt.title("Checkout Conversion Rate by Variant (with 95% CI)", fontsize=13, fontweight='bold')
plt.ylabel("Conversion Rate")
plt.ylim(0, 0.16)
plt.show()

4.2 Statistical Significance via Two-Proportion Z-Test

python
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep
 
count = np.array([summary.loc['Variant', 'Conversions'], summary.loc['Control', 'Conversions']])
nobs = np.array([summary.loc['Variant', 'Visitors'], summary.loc['Control', 'Visitors']])
 
# Run Two-Proportion Z-Test
z_stat, p_value = proportions_ztest(count=count, nobs=nobs, alternative='two-sided')
 
# Compute 95% Confidence Interval for the difference in proportions
ci_lower, ci_upper = confint_proportions_2indep(
    count1=count[0], nobs1=nobs[0],
    count2=count[1], nobs2=nobs[1],
    method='wald'
)
 
relative_lift = ((summary.loc['Variant', 'Conversion_Rate'] - summary.loc['Control', 'Conversion_Rate']) 
                 / summary.loc['Control', 'Conversion_Rate']) * 100
 
print(f"Z-Statistic:           {z_stat:.4f}")
print(f"P-Value:               {p_value:.6f}")
print(f"Measured Lift:         +{relative_lift:.2f}% relative")
print(f"95% CI for Lift Diff:  [{ci_lower * 100:.2f}%, {ci_upper * 100:.2f}%] absolute")
 
alpha = 0.05
if p_value < alpha:
    print("✅ DECISION: Reject Null Hypothesis. The Red Button provides statistically significant lift.")
else:
    print("❌ DECISION: Fail to Reject Null Hypothesis. Difference is within random noise.")
Z-Statistic:           3.7214
P-Value:               0.000198
Measured Lift:         +27.49% relative
95% CI for Lift Diff:  [1.24%, 4.01%] absolute
✅ DECISION: Reject Null Hypothesis. The Red Button provides statistically significant lift.

5. Scenario B: Revenue & Continuous Metrics (Welch's T-Test)

Conversion rate tests tell you if people buy; revenue tests tell you how much they spend. E-commerce revenue data is heavily right-skewed and exhibits unequal variances.

5.1 Simulating Continuous ARPU Data

python
np.random.seed(42)
n_users = 2500
 
# Control: Average Spend = $50, SD = $22
control_spend = np.random.gamma(shape=5.0, scale=10.0, size=n_users)
 
# Variant: Average Spend = $53, SD = $28 (Higher variance due to whale spenders)
variant_spend = np.random.gamma(shape=4.5, scale=11.8, size=n_users)
 
df_rev = pd.DataFrame({
    'Group': ['Control'] * n_users + ['Variant'] * n_users,
    'Spend': np.concatenate([control_spend, variant_spend])
})
 
print("Continuous Spend Summary by Group:")
print(df_rev.groupby('Group')['Spend'].describe().T[['count', 'mean', 'std', 'min', '50%', 'max']])

5.2 Welch's Two-Sample T-Test Implementation

Standard Student's t-test assumes equal variances ($\sigma_1^2 = \sigma_2^2$). In production, always set equal_var=False to execute Welch's T-Test:

python
from scipy import stats
 
control_values = df_rev[df_rev['Group'] == 'Control']['Spend']
variant_values = df_rev[df_rev['Group'] == 'Variant']['Spend']
 
# Execute Welch's T-Test
t_stat, p_val_ttest = stats.ttest_ind(variant_values, control_values, equal_var=False)
 
# Compute 95% Confidence Interval for the difference in means
diff_mean = variant_values.mean() - control_values.mean()
std_err = np.sqrt((control_values.var() / len(control_values)) + (variant_values.var() / len(variant_values)))
df_dof = len(control_values) + len(variant_values) - 2
t_crit = stats.t.ppf(0.975, df=df_dof)
 
ci_low = diff_mean - (t_crit * std_err)
ci_high = diff_mean + (t_crit * std_err)
 
print(f"Welch's T-Statistic:   {t_stat:.4f}")
print(f"P-Value:               {p_val_ttest:.5f}")
print(f"Mean Difference:       +${diff_mean:.2f} per user")
print(f"95% CI on Spend Diff:  [+${ci_low:.2f}, +${ci_high:.2f}]")
 
if p_val_ttest < 0.05:
    print("✅ DECISION: Statistically significant increase in Average Revenue Per User!")
else:
    print("❌ DECISION: No statistically significant difference in ARPU.")

6. Avoiding Critical Experimentation Pitfalls

                          COMMON A/B TESTING TRAPS
 1. THE PEEKING FALLACY         2. MULTIPLE TESTING (A/B/C/n)
 Checking daily p-values and    Testing 10 variants simultaneously
 stopping as soon as p < 0.05   inflates family-wise error to 40%+.
 inflates Type I error to >30%. Apply Bonferroni: α_adj = α / k.

 3. NOVELTY / PRIMACY EFFECT    4. DAY-OF-WEEK SEASONALITY
 Existing users temporarily     Running tests for 4 days ignores
 react to new UI before         weekend behavior shifts. Always
 reverting to baseline habit.   run full 7 or 14-day cycles.

6.1 The Peeking Problem (Continuous Monitoring)

If you check an experiment's p-value every morning and declare victory the first time $p < 0.05$, your true false positive rate is not 5% — it can exceed 30%.

Rule: Always commit to a fixed sample size determined before the test starts, and only evaluate the statistical test once that sample size is reached.

6.2 Multiple Testing Correction (Bonferroni)

If you test 3 different variants against 1 Control (A/B/C/D), the probability of finding at least one false positive at alpha = 0.05 is:

alpha_family = 1 - (1 - 0.05)^3 = 14.3%

Apply the Bonferroni Correction:

alpha_adjusted = alpha / k = 0.05 / 3 = 0.0167

Only declare variant B, C, or D a winner if its individual p-value is below 0.0167.

Complete Python A/B Testing Decision Engine

Here is a modular Python class you can drop into any analytics workflow:

python
import numpy as np
import pandas as pd
from scipy import stats
from statsmodels.stats.proportion import proportions_ztest
 
class ABTestEngine:
    def __init__(self, alpha: float = 0.05):
        self.alpha = alpha
 
    def analyze_proportion(self, control_conversions: int, control_total: int,
                           variant_conversions: int, variant_total: int) -> dict:
        """Evaluates binary conversion metrics using a 2-proportion Z-test."""
        p_c = control_conversions / control_total
        p_v = variant_conversions / variant_total
        lift = ((p_v - p_c) / p_c) * 100
 
        counts = np.array([variant_conversions, control_conversions])
        nobs = np.array([variant_total, control_total])
        z_stat, p_val = proportions_ztest(counts, nobs)
 
        return {
            'control_cr': p_c,
            'variant_cr': p_v,
            'relative_lift_pct': lift,
            'p_value': p_val,
            'statistically_significant': bool(p_val < self.alpha)
        }
 
    def analyze_means(self, control_data: pd.Series, variant_data: pd.Series) -> dict:
        """Evaluates continuous revenue/duration metrics using Welch's T-test."""
        t_stat, p_val = stats.ttest_ind(variant_data, control_data, equal_var=False)
        mean_c = control_data.mean()
        mean_v = variant_data.mean()
        lift = ((mean_v - mean_c) / mean_c) * 100
 
        return {
            'control_mean': mean_c,
            'variant_mean': mean_v,
            'relative_lift_pct': lift,
            'p_value': p_val,
            'statistically_significant': bool(p_val < self.alpha)
        }

Summary & What to Learn Next

Rigorous A/B testing transforms product development from opinions and politics into empirical science. By calculating sample sizes before launching, validating randomization integrity with SRM tests, choosing the correct statistical distribution test, and protecting against the peeking fallacy, you deliver trustworthy growth recommendations.

Next Steps:

Frequently Asked Questions

How do you calculate the minimum required sample size before starting an A/B test?

You compute sample size using statistical power analysis based on four parameters: baseline conversion rate (p1), Minimum Detectable Effect (MDE or p2), significance level (alpha = 0.05), and statistical power (1 - beta = 0.80). In Python, use statsmodels.stats.power.NormalIndPower.solve_power() with proportion_effectsize().

What is Sample Ratio Mismatch (SRM) and why does it invalidate an experiment?

SRM occurs when the actual proportion of users assigned to Control vs. Variant deviates significantly from the planned randomization ratio (e.g., 50/50). SRM indicates broken tracking, biased redirects, or bot filtering that systematically drops users from one variant, rendering any statistical significance invalid.

Why is the 'peeking problem' (continuous monitoring) dangerous in A/B testing?

Peeking means repeatedly checking p-values as daily data arrives and stopping the experiment early once p < 0.05. Because random noise causes p-values to fluctuate across time, continuous monitoring inflates the true Type I error rate (false positive rate) from 5% to over 30% unless sequential testing corrections are applied.

When should I use Welch's t-test instead of a standard Student's t-test?

Always prefer Welch's t-test (scipy.stats.ttest_ind with equal_var=False) when testing continuous revenue or session metrics. Welch's t-test does not assume equal variances between Control and Variant groups, protecting your analysis against variance heterogeneity caused by heavy spenders.

How do you choose between a Two-Proportion Z-Test and a Chi-Square test?

For two-variant binary conversion tests (Control vs Variant), the two-proportion z-test and Chi-square test yield mathematically identical p-values. Use the z-test when you need one-sided directional hypotheses or confidence intervals on lift; use Chi-square when analyzing multi-variant experiments (A/B/C/n).

Anuj Saini

Written by

Anuj SainiFounder & Lead Instructor

Founder at Topfolio with 6+ years in data & analytics across JPMC, Ultrahuman, and high-growth startups. Sat on hiring panels, reviewed 500+ resumes, and writes practical SQL & data guides.

View all articles

Related Articles

Tutorial

Product Analytics in Python: Funnel Conversion, Drop-off & User Journey Mapping

Master product funnel analysis in Python with Pandas and Plotly. Calculate step-by-step conversion rates, drop-off percentages, time-to-convert distributions, and interactive Sankey user journeys.

11 min readRead
Tutorial

Time Series Analysis & Forecasting in Python: Trend, Seasonality & Moving Averages

Complete guide to time series analysis and forecasting in Python. Master datetime indexing, resampling, moving average smoothing, seasonal decomposition, and the Augmented Dickey-Fuller (ADF) stationarity test.

12 min readRead
Tutorial

Customer Analytics in Python: Cohort Analysis, RFM Segmentation & LTV

Master customer analytics in Python with Pandas and Seaborn. Calculate RFM scores, build customer segments, run monthly cohort retention heatmaps, and estimate Customer Lifetime Value (LTV).

13 min readRead