A/B Testing &
Product Experimentation Analysis
A product team ran a 50,000-user A/B test on a new homepage design and reports a +13% conversion lift (p=0.015). The CPO asks: 'Should we ship this to 100% of users?' You'll audit the experiment end-to-end: check Sample Ratio Mismatch, measure 95% confidence bounds against MDE, uncover device segment anomalies (mobile friction), verify revenue guardrail stability with Welch's t-test, and author a definitive 1-page executive decision memo.
What you'll learn
- —Deduplicate raw clickstream event logs and isolate step-by-step conversion drop-offs
- —Construct triangular cohort retention matrices and visualize decay with Seaborn heatmaps
- —Evaluate binary conversion rates with two-sample proportion Z-tests and 95% Wald CIs
- —Verify revenue guardrail stability with Welch's two-sample T-test (equal_var=False)
- —Audit experimental integrity using Chi-square goodness-of-fit for Sample Ratio Mismatch (SRM)
- —Detect temporal novelty effect decay across multi-week experiment windows
- —Structure and defend a 1-page executive decision memo using the Minto SCR framework
The 5 milestones
Each milestone is reviewed before you advance.
- 01
Set up your project repo
Before any analytical work: create a public GitHub repository, push the standard skeleton, and paste the URL into your workspace.
- 02
Multi-Step Funnel & Friction Analysis
Ingest user clickstream logs, deduplicate multi-session events, compute step-over-step conversions, and isolate mobile drop-off friction.
- 03
Monthly Cohort Retention & Lifecycle Decay
Construct triangular monthly cohort retention matrices and visualize user lifecycle stabilization using Seaborn heatmaps.
- 04
A/B Test Statistical Evaluation & Diagnostics
Ingest 50k experiment rows, compute two-sample proportion Z-tests, 95% Wald CIs, Welch's ARPU tests, Sample Ratio Mismatch checks, and novelty fade.
- 05
Executive Decision Memo & Portfolio Deliverable
Synthesize findings into a high-stakes 1-page CPO executive recommendation memo and assemble your polished GitHub portfolio repository.
Reading & references
Build your portfolio & get certified
Want more projects like this? Join our Data Analyst Work Experience Program to complete 5+ guided industry projects, gain real experience, and earn an internship certificate.