When to Test, When Not To

# When to Test, When Not To A/B testing is the most over-used and under-thought tool in product analytics. ## Experiment feasibility calculation in Python ```python def calculate_test_duration(daily_traffic, baseline_cvr, mde_pp, power=0.80, alpha=0.05): # Calculates required days to run an A/B test without under-powering p = baseline_cvr mde_abs = mde_pp # Standard pocket formula for sample size per arm n_per_arm = 16 * p * (1 - p) / (mde_abs ** 2) total_n = n_per_arm * 2 days_required = total_n / daily_traffic return int(total_n), round(days_required, 1) # Scenario: 2,000 visitors/day, 4.0% baseline, target MDE = +0.8pp (0.008) n_needed, days = calculate_test_duration(daily_traffic=2000, baseline_cvr=0.04, mde_pp=0.008) print(f"Total Users Required: {n_needed:,} ({n_needed//2:,} per arm)") print(f"Estimated Duration: {days} days ({days/7:.1f} weeks)") if days > 28: print("⚠️ Test takes > 4 weeks. Consider increasing MDE or testing higher-traffic surface.") ``` ## Takeaway If the required runtime exceeds practical limits or leadership has already committed to shipping, use a phased observability rollout instead.