A/B Testing in Python: From Sample Size to p-Value Without the Ritual

Shipping the red button because it "felt higher" is guessing. This playbook runs the full experiment chain — simulation, SRM check, chi-square for conversions, and t-test for revenue — on data generated by courses/workbooks/generators/ab_testing.py.

What does rigorous A/B testing actually require?

A valid test needs four gates: well-formed randomisation, SRM check, correct test choice, and honest interpretation. Miss one and the p-value is theatre. The notebook isolates two scenarios so you practice both branches. See also the Product Funnels playbook for where experiments start, and the SQL window functions guide if you pull experiment exposures from a warehouse.

Ingredients: 4,000 simulated users (2,000 control blue, 2,000 variant red) at 10% vs 12% conversion for Scenario A, plus 2,000 revenue draws (mean 50->52) for Scenario B. Generator seed is 42 for reproducibility.

How do you simulate and sanity-check an experiment?

Every run starts with the same setup.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
 
sns.set_theme(style='whitegrid')

Generate Scenario A (binary conversion):

np.random.seed(42)
n_samples = 2000
 
control_conversions = np.random.binomial(n=1, p=0.10, size=n_samples)
control_df = pd.DataFrame({'Group': 'Control', 'Converted': control_conversions})
 
variant_conversions = np.random.binomial(n=1, p=0.12, size=n_samples)
variant_df = pd.DataFrame({'Group': 'Variant', 'Converted': variant_conversions})
 
df_ab = pd.concat([control_df, variant_df])
df_ab.sample(5)

Rendered output: five random rows of Group / Converted (0/1), confirming the long-format contract every later cell expects. The next check is SRM — df_ab['Group'].value_counts() must read 2000/2000; a chi-square goodness-of-fit against 50/50 outside p < 0.001 would halt analysis.

How do you test a conversion lift (Scenario A)?

Aggregate, visualise uncertainty, then test.

results = df_ab.groupby('Group').agg({'Converted': ['count','sum','mean']})
results.columns = ['Total Samples','Conversions','Conversion Rate']
results['Conversion Rate'] = results['Conversion Rate'] * 100
print(results)

Rendered output: a 2-row table — Control ~10.3% and Variant ~11.8% in one seed run (jitters ~1 pp).

plt.figure(figsize=(8,5))
sns.barplot(x='Group', y='Converted', data=df_ab, errorbar=('ci', 95), palette='pastel')
plt.title('Conversion Rate by Group (with 95% Confidence Interval)')
plt.ylabel('Conversion Rate')
plt.show()

Rendered output: two bars with overlapping 95% CI whiskers — the visual already warns that the lift may not survive testing.

contingency_table = pd.crosstab(df_ab['Group'], df_ab['Converted'])
chi2, p_value, dof, expected = stats.chi2_contingency(contingency_table)
print(f'P-Value: {p_value:.5f}')
alpha = 0.05
if p_value < alpha:
    print('Result is Statistically Significant! (Reject Null Hypothesis)')
else:
    print('Result is NOT Statistically Significant. (Fail to Reject Null)')

Rendered output on p~0.12: "Fail to Reject Null — any difference is likely due to chance." That is the correct call when CIs overlap; doubling N would narrow them.

How do you test a revenue lift (Scenario B)?

Continuous outcomes need a different engine.

control_spend = np.maximum(np.random.normal(loc=50, scale=20, size=1000), 0)
variant_spend = np.maximum(np.random.normal(loc=52, scale=25, size=1000), 0)
df_revenue = pd.DataFrame({
    'Spend': np.concatenate([control_spend, variant_spend]),
    'Group': ['Control']*1000 + ['Variant']*1000
})
print(df_revenue.groupby('Group')['Spend'].describe().round(2))

Rendered output: Control mean ~50.2, Variant ~52.1, with heavier right tail in Variant (higher SD). A KDE overlay makes the shift visual.

from scipy.stats import ttest_ind
stat, p = ttest_ind(
    df_revenue[df_revenue.Group=='Control'].Spend,
    df_revenue[df_revenue.Group=='Variant'].Spend,
    equal_var=False  # Welch — safer when SDs differ
)
print(f't={stat:.2f}, p={p:.4f}')

When do you ship and what do you report?

Ship only on: passed SRM, p < alpha and a meaningful lift inside the confidence interval. Report lift percentage, absolute delta, 95% CI, and N per arm — never p alone. Then replicate the decision in the Customer Analytics cohort to see if lift retains past week one.

Download the Notebook and Practise

This article is a walkthrough of a runnable Jupyter notebook. Download the original .ipynb and run it locally or on Colab — every code block above appears in order.

Continue your track: Data Analyst Roadmap · Python and Pandas Guide · SQL NULL Handbook · SQL JOIN Fan-Out · Topfolio Practice · Data Analyst vs Engineer

Dataset generators where applicable are in courses/workbooks/generators/ — see citations atop for the exact *.py source for this notebook.

Frequently Asked Questions

When do I use chi-square vs t-test for an A/B test?

Chi-square (or z-test for proportions) for binary outcomes like Converted (yes/no); t-test (scipy.stats.ttest_ind) for continuous outcomes like Spend or revenue per user.

What is SRM and why check it?

Sample Ratio Mismatch means the observed split (e.g., 48/52) diverges from the intended 50/50. It signals broken randomisation or logging. Always run a chi-square goodness-of-fit on group counts before reading lift.

Why does my p-value flip when I peek daily?

Repeated looks inflate false positives. Peeking without correction is p-hacking. Fix by pre-committing sample size or using sequential methods, not by re-running chi2_contingency until p < 0.05.

Is p=0.04 a business win?

Not alone. Report effect size and confidence interval alongside p: a 0.04 on a +0.2% lift with +/-1% CI is not actionable. Lift plus interval drives the ship decision.