This lesson on Why Experiments Go Wrong - Five Failure Modes is hands-on and example-driven. You will identify and resolve the five critical failure modes in online A/B testing: underpowered sample sizing, multiple testing false discovery accumulation, behavioral novelty and primacy effects, and unit interference in social networks or two-sided marketplaces. You will learn to size tests correctly, adjust significance thresholds, and select non-standard randomization units like clusters and geos.
What You'll Be Able To Do
- Calculate sample size requirements using statistical power, significance level, variance, and minimum detectable effect.
- Apply the Bonferroni correction and False Discovery Rate controls to mitigate cumulative Type I error in multi-variant tests.
- Diagnose novelty and primacy effects by isolating metric trajectories between new and existing cohorts over time.
- Identify interference mechanisms in social network spillover and marketplace resource competition.
- Design cluster-based, ego network, and geo-randomized experiments to isolate treatment units from spillover interference.
Detailed Concept Walkthrough
1. Sample Size and Test Duration Parameters
An experiment must collect sufficient sample size and run across complete business cycles to detect true effects while controlling false positives and false negatives.
- Core Parameters: Required sample size per variant is governed by significance level $\alpha$ (Type I error, typically 5%), statistical power $1 - \beta$ (Type II error, typically 80%), population variance $\sigma^2$, and the Minimum Detectable Effect $\delta$.
- Heuristic Formula: For standard 80% power and 5% significance, the per-variant sample size follows the rule of thumb $n \approx \frac{16 \cdot \sigma^2}{\delta^2}$. Smaller intended detectable effects require quadratic increases in sample volume.
- Temporal Duration: Regardless of achieving statistical sample size quickly due to high traffic, experiments must run for a minimum duration of 14 days. This captures complete day-of-week seasonality and cyclical user behavioral shifts.
# Sample size rule-of-thumb: n ≈ 16 * variance / delta^2
def estimate_sample_size(variance: float, mde: float) -> int:
return int(16 * variance / (mde ** 2))
# Example: baseline variance 0.25, minimum detectable effect 0.01 (1% absolute lift)
required_n = estimate_sample_size(variance=0.25, mde=0.01)
print(f"Required sample per variant: {required_n:,}")
Key Takeaway: Always run tests for at least two full weeks and scale sample sizes inversely with the square of the desired effect size.
2. The Multiple Testing Problem and Corrections
Evaluating multiple variants or metrics simultaneously inflates the cumulative probability of finding at least one false positive.
- Error Accumulation: When evaluating $k$ independent variants at significance level $\alpha$, the cumulative probability of at least one false positive is $\alpha_{cumulative} = 1 - (1 - \alpha)^k$. Testing 10 variants at $\alpha = 0.05$ produces an actual false discovery risk of $\approx 40.1%$.
- Bonferroni Correction: Controls Family-Wise Error Rate (FWER) by adjusting the significance threshold for each test to $\alpha_{adjusted} = \frac{\alpha}{k}$. While mathematically rigorous, it becomes overly conservative when testing dozens or hundreds of metrics.
- False Discovery Rate (FDR): Controls the expected proportion of false positives among all rejected null hypotheses. FDR methods (e.g., Benjamini-Hochberg) provide higher statistical power than Bonferroni when monitoring extensive metric dashboards.
# Calculate cumulative Type I error and apply Bonferroni correction
def multi_test_alpha(alpha: float, num_variants: int) -> dict:
cum_alpha = 1 - ((1 - alpha) ** num_variants)
bonferroni = alpha / num_variants
return {"cumulative_risk": round(cum_alpha, 4), "bonferroni_threshold": round(bonferroni, 4)}
# Example: 10 variants at standard alpha = 0.05
print(multi_test_alpha(alpha=0.05, num_variants=10))
Key Takeaway: Adjust alpha thresholds via Bonferroni or FDR whenever testing multiple variants or scanning multiple metric surfaces.
3. Novelty and Primacy Behavioral Biases
Initial metric movements following a product change often reflect temporary user adaptation patterns rather than permanent preference shifts.
- Novelty Effect: Existing users interact with a new feature simply because it is unfamiliar, causing an artificial short-term spike in engagement metrics that decays over time.
- Primacy Effect (Change Aversion): Existing users react negatively to disrupted habits and modified workflows, creating an initial drop in conversion or satisfaction metrics that recovers as users adapt.
- Diagnostic Strategy: Plot metric performance by cohort over time to identify stabilization trends. Isolate first-time users (who have no prior baseline and zero change aversion) from long-term power users to establish true steady-state impact.
# Diagnostic pseudo-code: Compare metric trend between new vs existing users
# SELECT
# user_cohort_type, -- 'first_time_user' vs 'existing_user'
# experiment_day, -- Day 1 to 28
# AVG(metric_value) AS avg_engagement
# FROM experiment_logs
# GROUP BY 1, 2 ORDER BY 2, 1;
Key Takeaway: Diagnose behavioral adaptation by tracking performance trends over time and isolating cohorts of first-time users.
4. Interference in Networks and Two-Sided Marketplaces
Standard user-level A/B testing assumes the Stable Unit Treatment Value Assumption (SUTVA), which fails when treatment units impact control units.
- Social Network Spillover: When a social platform rolls out a content creation feature to treatment users, their control friends consume that content, increasing control engagement and causing standard user-level tests to underestimate treatment impact.
- Marketplace Resource Competition: In two-sided platforms (e.g., Uber, Lyft, Airbnb), treatment driver/rider incentives consume a shared pool of physical resources. Giving vouchers to treatment riders depletes available drivers for control riders, overestimating treatment lift.
- Mitigation Architecture: Eliminate interference by grouping dependent units together. Implement cluster-based randomization for graph networks, ego network randomization for local social clusters, time-based switchbacks, or geo-based isolation for physical supply-demand markets.
# Example strategy mapping for interference mitigation
def select_experiment_design(has_network_effects: bool, is_shared_marketplace: bool) -> str:
if is_shared_marketplace:
return "Geo-based randomization or Time-based switchback"
if has_network_effects:
return "Cluster-based or Ego-network randomization"
return "Standard user-level A/B test"
Key Takeaway: When units share a network or supply pool, switch from user-level randomization to cluster, geo, or time-based designs.
Topics Covered in Why Experiments Go Wrong - Five Failure Modes
- A/B Testing Fundamentals (01:20 - 02:32) — Covers the definition of controlled experiments, treatment groups, and hypothesis testing foundations.
- Duration and Sample Size (02:32 - 04:47) — Explains sample sizing heuristics and the necessity of two-week runtimes for cyclicality.
- Multiple Testing Pitfalls (04:47 - 07:47) — Demonstrates cumulative false positive accumulation and how to apply Bonferroni and FDR corrections.
- Novelty and Primacy Effects (07:47 - 09:39) — Details user behavioral biases after launch and methods to isolate true steady-state engagement.
- Unit Interference Mechanics (09:39 - 12:43) — Explores how network spillover and marketplace supply competition violate experimental isolation assumptions.
- Interference Mitigation Architectures (12:43 - 15:42) — Introduces geo-based, switchback, cluster, and ego-network randomization techniques to isolate interference.
Stats & Product Analytics for Analysts Cheat Sheet
-
Sample Size Heuristic— Approximates required sample size per variant for standard powern_per_variant = 16 * variance / (mde ** 2) -
Bonferroni Threshold— Calculates corrected per-test significance level across multiple variantsalpha_bonferroni = alpha / num_comparisons -
Cumulative Alpha Risk— Computes aggregate false positive probability across multiple independent testscumulative_type1_error = 1 - ((1 - alpha) ** k) -
Geo-Based Randomization— Assigns entire geographic markets to treatment or controlvariant = 'treatment' if city_id in treatment_cities else 'control' -
Ego Network Cluster— Assigns central user and primary connection cluster to same treatmentcluster_variant = assign_variant(ego_user_id, salt='exp_v1')
Comparison Table
| Failure Mode | Root Mechanism | Correction Strategy |
|---|---|---|
| Multiple Testing | Type I error accumulation across variants | Bonferroni or False Discovery Rate |
| Novelty / Primacy | Transient curiosity or change aversion | First-time user cohort segmentation |
| Network Spillover | Treatment actions bleed to control friends | Cluster or ego network randomization |
| Marketplace Competition | Treatment consumes shared supply pool | Geo-based or time-switchback randomization |
Common Pitfalls
- Mistake: Stopping an experiment early once statistical significance is reached. Avoid: Running tests for a full 14-day duration to prevent sample bias and capture cyclical trends.
- Mistake: Using standard user-level randomization in two-sided supply markets. Avoid: Using geo-based or time-switchback randomization to prevent treatment groups from starving control supply.
- Mistake: Relying on Bonferroni adjustments when testing hundreds of secondary dashboard metrics. Avoid: Using False Discovery Rate corrections to prevent excessive false negatives on large metric sets.
FAQs
- Why does the sample size formula scale with the inverse square of MDE? Because standard error decreases with the square root of sample size, detecting half as large an effect requires quadrupling the total sample volume.
- How can you differentiate a novelty effect from genuine product improvement? Plot performance over time; novelty effects display an initial spike followed by exponential decay, whereas genuine improvements maintain a stabilized lift.
- Why do network effects underestimate treatment effect in standard user-level A/B tests? Treatment user engagement spills over to control friends, raising control metric baselines and artificially shrinking the measured delta.