SUBMISSION: Project 2 — End-to-End A/B Experiment Analysis with Sample Sizing & Welch's T-Test

🎯 Hands-on Portfolio Deliverable. The single most-interviewed DS workflow: "walk me through an A/B test end to end."

🏢 Business Scenario & Executive Prompt

ShopKart (fashion e-commerce, 10K visitors/day) redesigned its homepage hero. The PM pre-registered conversion rate as the primary metric (baseline 4.0%) but also instrumented revenue/visitor, mobile conversion, desktop conversion, and 6 country slices. After 10 days the dashboard flashes "mobile +3.2%, p = 0.04 — SHIP IT?" The growth lead wants to ship today. You are asked for the statistically honest answer — and the sizing math for the next test.

📋 Source Data & Input Schemas

shopkart_ab.csv — 100,000 rows (50K/arm, mock shape; scaffold generates it):

Stakes: 1% absolute lift = ~₹45L incremental annual GTV. A false ship costs ~2 sprints of eng + polluted learnings.

🎯 Required Deliverables

1. ab_analysis.ipynb — sizing → analysis → guardrails notebook.
2. ab_results.csv — one row per metric/slice: metric, slice, control, variant, lift, ci_lo, ci_hi, p_value, significant_bonferroni.
3. ship_memo.md — ship / don't-ship verdict with the multiple-testing diagnosis and next-test sizing.
4. power_curve.png — required-n-per-arm vs MDE curve for baseline 4%.

🛠️ Step-by-Step Implementation Guide

1. Size before you look. Two-proportion power formula for 1% absolute lift (4%→5%), α=0.05, power 0.8 → n/arm. Plot the power curve for MDEs 0.5%–3%.
2. Primary analysis. Difference in conversion proportions + 95% CI; Welch's t-test on revenue_inr (unequal variances — justify Welch over pooled with a variance check).
3. Slice autopsy. Recompute the "mobile p=0.04" claim across all 1 + 1 + 2 + 6 = 10 looks; Bonferroni + BH-FDR correction; report which (if any) survive.
4. Peeking audit. Show how a day-5 peek at p≈0.06 would have misled; state the sequential-testing-correct alternative in one paragraph.
5. Verdict. Ship / iterate / kill with expected-value math (lift × traffic × margin vs eng cost).

💻 Production Starter Scaffold

import numpy as np, pandas as pd
from scipy import stats
from statsmodels.stats.proportion import confint_proportions_2indep
from statsmodels.stats.multitest import multipletests

rng = np.random.default_rng(7)
N = 50_000
def arm(n, conv, rev_mu=6.2, rev_sd=1.0):
    c = rng.binomial(1, conv, n)
    return pd.DataFrame({"converted": c,
        "revenue_inr": np.where(c, np.round(rng.lognormal(rev_mu, rev_sd, n),2), 0.0)})
ctrl = arm(N, 0.040); var = arm(N, 0.040)  # A/A under the hood: no true effect
ctrl["arm"]=0; var["arm"]=1
df = pd.concat([ctrl,var], ignore_index=True)
df["user_id"]=[f"U-{i:06d}" for i in range(len(df))]
df["device"]=rng.choice(["mobile","desktop"],len(df),p=[.65,.35])
df["country"]=rng.choice(["IN-MH","IN-DL","IN-KA","IN-TN","IN-WB","IN-OT"],len(df))

# --- sizing: two-proportion n per arm ---
p1, p2, za, zb = 0.04, 0.05, 1.96, 0.84
p_bar=(p1+p2)/2
n = ((za+zb)*np.sqrt(2*p_bar*(1-p_bar)))**2/(p2-p1)**2
print("required n/arm:", round(n))

# --- primary + Welch on revenue ---
a=df[df.arm==1]; b=df[df.arm==0]
lo,hi = confint_proportions_2indep(a.converted.sum(),len(a),b.converted.sum(),len(b),compare='diff')
print("lift CI:", round(lo,4), round(hi,4))
print("Welch revenue:", stats.ttest_ind(a.revenue_inr,b.revenue_inr,equal_var=False))
print("var ratio:", a.revenue_inr.var()/b.revenue_inr.var())

🧪 Acceptance Criteria & Verification Checks

• power_curve.png shows n/arm exploding as MDE shrinks (inverse-square shape visible).
• Primary conversion lift CI covers 0 (A/A data) and memo refuses to ship on the mobile slice.
• ab_results.csv has Bonferroni + FDR columns; ≥10 rows (primary + slices).
• Welch (not pooled) used for revenue with a variance-ratio justification printed.
• Memo includes next-test sizing (n/arm + calendar days at 10K/day).

📊 Evaluation Rubric (100 points — passing ≥ 80)

📤 Submission Guidelines

1. Upload your files (.ipynb, .csv, .md, .png) via the file dropzone on this lesson.
2. Paste your public GitHub repo URL in the submission notes (repo must contain the same files + a 5-line README with the run command).
3. Name files exactly as listed under Required Deliverables — autograding checks filenames first.
4. Passing score is 80/100. You may resubmit; the highest score counts.