Why this lesson exists
Right after this course, this milestone's graded Project asks you to analyze a real 50,000-user A/B test and write an executive memo - SRM audit, lift + CI, guardrail check, segment anomaly, novelty-decay, the works. That's the real deliverable, at real stakes.
This lesson is the rep before the rep that counts. Small dataset, fully specified, every check has an expected-answer range so you can tell yourself whether you got it right - no guessing, no waiting for a grader. Everything you do here, you'll do again - at higher stakes, on a real 50K-user dataset - in the Milestone Project that follows.
The scenario
A mid-size SaaS company redesigned its homepage hero section (new headline + CTA button) and ran a 14-day, 50/50 A/B test on it. Traffic was modest - about 500 visitors total landed on the homepage during the window and were bucketed into Control (old hero) or Treatment (new hero). The primary metric is signup started (did the visitor click through to the signup form). A pre-registered MDE of +4.0 percentage points was agreed with the PM before the test launched - anything smaller isn't worth the engineering cost of keeping the new hero.
Here's what the underlying ~500-row event log looks like (one row per visitor):
| date | arm | device | started_signup |
|---|---|---|---|
| 2026-06-01 | control | desktop | 0 |
| 2026-06-01 | treatment | mobile | 1 |
| 2026-06-01 | control | mobile | 0 |
| 2026-06-02 | treatment | desktop | 1 |
| ... | ... | ... | ... (496 more rows) |
You won't work row-by-row. Below are the pre-aggregated numbers you need for each check - exactly the numbers you'd get by grouping that CSV in pandas.
Your 5 tasks
Work through these in order - they're the same 5 checks from "Your Analyst Checklists," applied to real numbers. Each task gives you an expected range; if your answer lands outside it, you made an arithmetic or methodology slip - go back and check your test/formula choice before moving on.
Task 1 - SRM check
Total visitors: Control = 244, Treatment = 256 (out of 500; target split was 50/50).
Run a chi-square goodness-of-fit test against the expected 250/250 split (same code as the Five Failure Modes tutorial).
Expected range: χ² between 0.2 and 0.4, p-value between 0.5 and 0.7. Conclusion: no SRM detected - the small skew is within normal randomization noise.
Task 2 - Primary lift + 95% CI
- Control: 25 conversions / 244 visitors
- Treatment: 43 conversions / 256 visitors
Compute the two-proportion z-test lift, p-value, and 95% CI (same code as "Anatomy of an A/B Test Result").
Expected range: lift between +5.5pp and +7.5pp (relative lift roughly +55% to +75%), p-value between 0.02 and 0.05, 95% CI lower bound between 0% and +2%, upper bound between +11% and +14%.
Task 3 - MDE bar check
The pre-registered MDE was +4.0pp.
Expected range: your Task 2 lift clears the MDE bar comfortably (lift ≥ 4.0pp with the CI lower bound above 0). On these two checks alone, this looks like a clean ship.
Task 4 - Guardrail metric check
Guardrail: average session duration (are we just attracting less-invested clickers, not better prospects?).
- Control: mean = 38.2s, std = 14.5s, n = 244
- Treatment: mean = 36.9s, std = 15.1s, n = 256
Run Welch's t-test on the difference in means.
Expected range: t-statistic between -1.5 and -0.5. Not statistically significant (|t| < 1.96) - guardrail holds, no detected harm.
Task 5 - Novelty-fade check
Split the same conversions by week:
| Control conversions / n | Treatment conversions / n | |
|---|---|---|
| Week 1 (days 1-7) | 8 / 122 | 27 / 128 |
| Week 2 (days 8-14) | 17 / 122 | 16 / 128 |
Compute the lift separately for each week (same code as the Five Failure Modes tutorial's novelty check).
Expected range: Week 1 lift between +12pp and +17pp. Week 2 lift between -4pp and +2pp (statistically indistinguishable from zero - it may even come out slightly negative).
Put it together
Notice what just happened: Task 2's headline lift (+6.5pp-ish) and Task 3's MDE check both said ship it. Task 5 says something very different - almost the entire lift happened in week 1 and had faded to roughly nothing by week 2. That's a textbook novelty effect: users clicked the new CTA because it was new, not because it's durably better.
Write 3-4 sentences (paste into the activity log, no upload needed) answering: given all 5 checks together, would you ship this treatment as-is? What would you tell the PM to do instead - ship, kill, or extend the test?
Takeaway
A checklist only works if you run every item on it. Two out of five green checks (lift + MDE) felt like a "ship it" story - a fifth check (novelty fade) turned it into a "not yet" story. This is exactly the discipline the Milestone Project grades you on, just at 100x the row count and real stakes. Go do it for real next.