Micro Free Course — 100% Free Learning
All lessons in this module are free to learn. Sign in with Google to save your progress.
A/B Testing in Practice
When to test, how to design the test, sample-size napkin math, reading results, and the five failure modes that ruin experiments.
Module Content
Why A/B Test At All
Sometimes experiments are wasted effort. Sometimes they're theatre. An ex-Meta PM walks through the five decisions that separate a real experiment from testing theatre - primary vs. guardrail metrics, and pre-committing to ship/no-ship/retest before you look at results. FIND_VIDEO: search 'Dianna Yau 5 steps before running a product A/B test experiment' - recommended channel: Dianna Yau. Aim for 11 min or under.
When to Test, When Not To
A/B tests are expensive in time and traffic. Spend them on decisions where the result will actually change what you do.
When to Test, When Not To Quiz
Test what you just read.
Designing an A/B Test - Hypothesis to Guardrails
Hypothesis → primary metric → guardrails → exposure unit. Get any of these wrong and the test is invalid. FIND_VIDEO: search 'Emma Ding AB Testing Fundamentals What Every Data Scientist Needs to Know' - recommended channel: Emma Ding. Aim for 10 min or under.
The Test Design Checklist
Most A/B test failures are design failures, not analysis failures. Use this 6-point pre-flight check.
The Test Design Checklist Quiz
Test what you just read.
Sample Size & Minimum Detectable Effect - The Napkin Version
Build sample-size intuition without formulas. Why a 1% lift needs a much bigger sample than 10%. FIND_VIDEO: search 'ab test sample size minimum detectable effect intuition' - recommended channel: Emma Ding. Aim for 10 min or under.
Sample-Size Napkin Math
A rough sample-size formula and the intuitions that come with it. Lets you sanity-check any test plan in your head.
Sample-Size Napkin Math Quiz
Test what you just read.
Reading A/B Test Results
Decoding the output of common experimentation platforms - what to look at, what to ignore. FIND_VIDEO: search 'Emma Ding AB Testing Made Easy Real Life Example' - recommended channel: Emma Ding. Aim for 10 min or under.
Anatomy of an A/B Test Result
Reading a real test dashboard. The five numbers that matter and the ten that don't.
Anatomy of an A/B Test Result Quiz
Test what you just read.
Why Experiments Go Wrong - Five Failure Modes
Peeking, novelty effects, contamination, network effects, segment surprises - the five ways tests lie. FIND_VIDEO: search 'Emma Ding Crack AB Testing Problems for Data Science Interviews' - recommended channel: Emma Ding. Aim for 12 min or under.
The Five Failure Modes
Even well-designed tests fail in predictable ways. Memorize the five.
The Five Failure Modes Quiz
Test what you just read.