Warm-Up Rep - Practice Before the Milestone Project

# Warm-Up Rep - Practice Before the Milestone Project ## Why this lesson exists Right after this course, this milestone's graded Project asks you to analyze a **real 50,000-user A/B test** and write an executive memo - SRM audit, lift + CI, guardrail check, segment anomaly, novelty-decay, the works. That's the real deliverable, at real stakes. This lesson is the rep before the rep that counts. Small dataset, fully specified, every check has an expected-answer range so you can tell yourself whether you got it right - no guessing, no waiting for a grader. **Everything you do here, you'll do again - at higher stakes, on a real 50K-user dataset - in the Milestone Project that follows.** ## The scenario A mid-size SaaS company redesigned its homepage hero section (new headline + CTA button) and ran a 14-day, 50/50 A/B test on it. Traffic was modest - about 500 visitors total landed on the homepage during the window and were bucketed into Control (old hero) or Treatment (new hero). The primary metric is **signup started** (did the visitor click through to the signup form). A pre-registered MDE of **+4.0 percentage points** was agreed with the PM before the test launched - anything smaller isn't worth the engineering cost of keeping the new hero. Here's what the underlying ~500-row event log looks like (one row per visitor): | date | arm | device | started_signup | |---|---|---|---| | 2026-06-01 | control | desktop | 0 | | 2026-06-01 | treatment | mobile | 1 | | 2026-06-01 | control | mobile | 0 | | 2026-06-02 | treatment | desktop | 1 | | ... | ... | ... | ... (496 more rows) | You won't work row-by-row. Below are the pre-aggregated numbers you need for each check - exactly the numbers you'd get by grouping that CSV in pandas. ## Your 5 tasks Work through these in order - they're the same 5 checks from "Your Analyst Checklists," applied to real numbers. Each task gives you an **expected range**; if your answer lands outside it, you made an arithmetic or methodology slip - go back and check your test/formula choice before moving on. ### Task 1 - SRM check Total visitors: **Control = 244, Treatment = 256** (out of 500; target split was 50/50). Run a chi-square goodness-of-fit test against the expected 250/250 split (same code as the Five Failure Modes tutorial). **Expected range:** χ² between 0.2 and 0.4, p-value between 0.5 and 0.7. Conclusion: **no SRM detected** - the small skew is within normal randomization noise. ### Task 2 - Primary lift + 95% CI - Control: **25 conversions / 244 visitors** - Treatment: **43 conversions / 256 visitors** Compute the two-proportion z-test lift, p-value, and 95% CI (same code as "Anatomy of an A/B Test Result"). **Expected range:** lift between +5.5pp and +7.5pp (relative lift roughly +55% to +75%), p-value between 0.02 and 0.05, 95% CI lower bound between 0% and +2%, upper bound between +11% and +14%. ### Task 3 - MDE bar check The pre-registered MDE was **+4.0pp**. **Expected range:** your Task 2 lift clears the MDE bar comfortably (lift ≥ 4.0pp with the CI lower bound above 0). On these two checks alone, this looks like a clean ship. ### Task 4 - Guardrail metric check Guardrail: **average session duration** (are we just attracting less-invested clickers, not better prospects?). - Control: mean = 38.2s, std = 14.5s, n = 244 - Treatment: mean = 36.9s, std = 15.1s, n = 256 Run Welch's t-test on the difference in means. **Expected range:** t-statistic between -1.5 and -0.5. **Not statistically significant** (|t| < 1.96) - guardrail holds, no detected harm. ### Task 5 - Novelty-fade check Split the same conversions by week: | | Control conversions / n | Treatment conversions / n | |---|---|---| | Week 1 (days 1-7) | 8 / 122 | 27 / 128 | | Week 2 (days 8-14) | 17 / 122 | 16 / 128 | Compute the lift separately for each week (same code as the Five Failure Modes tutorial's novelty check). **Expected range:** Week 1 lift between +12pp and +17pp. Week 2 lift between -4pp and +2pp (statistically indistinguishable from zero - it may even come out slightly negative). ## Put it together Notice what just happened: Task 2's headline lift (+6.5pp-ish) and Task 3's MDE check both said **ship it**. Task 5 says something very different - almost the entire lift happened in week 1 and had faded to roughly nothing by week 2. That's a textbook novelty effect: users clicked the new CTA because it was new, not because it's durably better. Write 3-4 sentences (paste into the activity log, no upload needed) answering: **given all 5 checks together, would you ship this treatment as-is?** What would you tell the PM to do instead - ship, kill, or extend the test? ## Takeaway A checklist only works if you run every item on it. Two out of five green checks (lift + MDE) felt like a "ship it" story - a fifth check (novelty fade) turned it into a "not yet" story. This is exactly the discipline the Milestone Project grades you on, just at 100x the row count and real stakes. G