SUBMISSION: Project 2 — End-to-End A/B Experiment Analysis with Sample Sizing & Welch's T-Test

# SUBMISSION: Project 2 — End-to-End A/B Experiment Analysis with Sample Sizing & Welch's T-Test > 🎯 **Hands-on Portfolio Deliverable.** The single most-interviewed DS workflow: *"walk me through an A/B test end to end."* ## 🏢 Business Scenario & Executive Prompt **ShopKart** (fashion e-commerce, 10K visitors/day) redesigned its homepage hero. The PM pre-registered **conversion rate** as the primary metric (baseline 4.0%) but also instrumented revenue/visitor, mobile conversion, desktop conversion, and 6 country slices. After 10 days the dashboard flashes *"mobile +3.2%, p = 0.04 — SHIP IT?"* The growth lead wants to ship today. You are asked for the statistically honest answer — and the sizing math for the *next* test. ## 📋 Source Data & Input Schemas **`shopkart_ab.csv`** — 100,000 rows (50K/arm, mock shape; scaffold generates it): | column | dtype | description | |---|---|---| | `user_id` | str | unique visitor | | `arm` | 0/1 | 0 = control, 1 = variant | | `converted` | 0/1 | purchased within session | | `revenue_inr` | float | revenue (0 if not converted; log-normal if converted) | | `device` | category | `mobile \| desktop` | | `country` | category | 6 slices (`IN-MH`, `IN-DL`, `IN-KA`, `IN-TN`, `IN-WB`, `IN-OT`) | Stakes: 1% absolute lift = ~₹45L incremental annual GTV. A false ship costs ~2 sprints of eng + polluted learnings. ## 🎯 Required Deliverables 1. **`ab_analysis.ipynb`** — sizing → analysis → guardrails notebook. 2. **`ab_results.csv`** — one row per metric/slice: `metric, slice, control, variant, lift, ci_lo, ci_hi, p_value, significant_bonferroni`. 3. **`ship_memo.md`** — ship / don't-ship verdict with the multiple-testing diagnosis and next-test sizing. 4. **`power_curve.png`** — required-n-per-arm vs MDE curve for baseline 4%. ## 🛠️ Step-by-Step Implementation Guide 1. **Size before you look.** Two-proportion power formula for 1% absolute lift (4%→5%), α=0.05, power 0.8 → n/arm. Plot the power curve for MDEs 0.5%–3%. 2. **Primary analysis.** Difference in conversion proportions + 95% CI; Welch's t-test on `revenue_inr` (unequal variances — justify Welch over pooled with a variance check). 3. **Slice autopsy.** Recompute the "mobile p=0.04" claim across all 1 + 1 + 2 + 6 = 10 looks; Bonferroni + BH-FDR correction; report which (if any) survive. 4. **Peeking audit.** Show how a day-5 peek at p≈0.06 would have misled; state the sequential-testing-correct alternative in one paragraph. 5. **Verdict.** Ship / iterate / kill with expected-value math (lift × traffic × margin vs eng cost). ## 💻 Production Starter Scaffold ```python import numpy as np, pandas as pd from scipy import stats from statsmodels.stats.proportion import confint_proportions_2indep from statsmodels.stats.multitest import multipletests rng = np.random.default_rng(7) N = 50_000 def arm(n, conv, rev_mu=6.2, rev_sd=1.0): c = rng.binomial(1, conv, n) return pd.DataFrame({"converted": c, "revenue_inr": np.where(c, np.round(rng.lognormal(rev_mu, rev_sd, n),2), 0.0)}) ctrl = arm(N, 0.040); var = arm(N, 0.040) # A/A under the hood: no true effect ctrl["arm"]=0; var["arm"]=1 df = pd.concat([ctrl,var], ignore_index=True) df["user_id"]=[f"U-{i:06d}" for i in range(len(df))] df["device"]=rng.choice(["mobile","desktop"],len(df),p=[.65,.35]) df["country"]=rng.choice(["IN-MH","IN-DL","IN-KA","IN-TN","IN-WB","IN-OT"],len(df)) # --- sizing: two-proportion n per arm --- p1, p2, za, zb = 0.04, 0.05, 1.96, 0.84 p_bar=(p1+p2)/2 n = ((za+zb)*np.sqrt(2*p_bar*(1-p_bar)))**2/(p2-p1)**2 print("required n/arm:", round(n)) # --- primary + Welch on revenue --- a=df[df.arm==1]; b=df[df.arm==0] lo,hi = confint_proportions_2indep(a.converted.sum(),len(a),b.converted.sum(),len(b),compare='diff') print("lift CI:", round(lo,4), round(hi,4)) print("Welch revenue:", stats.ttest_ind(a.revenue_inr,b.revenue_inr,equal_var=False)) print("var ratio:", a.revenue_inr.var()/b.revenue_inr.var()) ``` ## 🧪 Acceptance Criteria & Verification Checks - [ ] `power_curve.png` shows n/arm exploding as MDE shrinks (inverse-square shape visible). - [ ] Primary conversion lift CI covers 0 (A/A data) and memo refuses to ship on the mobile slice. - [ ] `ab_results.csv` has Bonferroni + FDR columns; ≥10 rows (primary + slices). - [ ] Welch (not pooled) used for revenue with a variance-ratio justification printed. - [ ] Memo includes next-test sizing (n/arm + calendar days at 10K/day). --- ## 📊 Evaluation Rubric (100 points — passing ≥ 80) | # | Criterion | Points | What earns full marks | |---|---|---|---| | 1 | Statistical correctness | 25 | Right distributions/tests/CIs; assumptions checked, not assumed; math is reproducible | | 2 | Code quality & reproducibility | 25 | Seeded RNG, clean re-runnable notebook, typed schemas, no hardcoded magic numbers | | 3 | Business reasoning & memo | 25 | Decision-oriented memo with costed trade-offs; honest about uncertainty and limits | | 4 | Visualization & communication | 25 | Labeled, honest axe