When to Test, When Not To

A/B testing is the most over-used and under-thought tool in product analytics.

Experiment feasibility calculation in Python

def calculate_test_duration(daily_traffic, baseline_cvr, mde_pp, power=0.80, alpha=0.05):
    # Calculates required days to run an A/B test without under-powering
    p = baseline_cvr
    mde_abs = mde_pp
    # Standard pocket formula for sample size per arm
    n_per_arm = 16 * p * (1 - p) / (mde_abs ** 2)
    total_n = n_per_arm * 2
    days_required = total_n / daily_traffic
    return int(total_n), round(days_required, 1)

# Scenario: 2,000 visitors/day, 4.0% baseline, target MDE = +0.8pp (0.008)
n_needed, days = calculate_test_duration(daily_traffic=2000, baseline_cvr=0.04, mde_pp=0.008)
print(f"Total Users Required: {n_needed:,} ({n_needed//2:,} per arm)")
print(f"Estimated Duration:   {days} days ({days/7:.1f} weeks)")
if days > 28:
    print("⚠️ Test takes > 4 weeks. Consider increasing MDE or testing higher-traffic surface.")

Takeaway

If the required runtime exceeds practical limits or leadership has already committed to shipping, use a phased observability rollout instead.