Which Error Costs You More?

# Which Error Costs You More? Two errors are possible in any hypothesis test: - **Type I error (false positive)** - rejecting the null when it's actually true. You ship a treatment that doesn't actually work. - **Type II error (false negative)** - failing to reject the null when the alternative is actually true. You DON'T ship a treatment that does work. ## Calculating Statistical Power in Python ```python import numpy as np from statsmodels.stats.power import NormalIndPower from statsmodels.stats.proportion import proportion_effectsize # Baseline conversion = 10% (0.10), Expected Treatment = 11.5% (0.115) -> MDE = 1.5pp p1 = 0.10 p2 = 0.115 effect_size = proportion_effectsize(p1, p2) power_analyzer = NormalIndPower() # Calculate power for N = 5,000 per arm at alpha = 0.05 n_sample = 5000 alpha = 0.05 achieved_power = power_analyzer.solve_power( effect_size=effect_size, nobs1=n_sample, alpha=alpha, power=None, ratio=1.0 ) print(f"Effect size (Cohen's h): {effect_size:.4f}") print(f"Sample per arm: {n_sample:,}") print(f"Statistical Power (1-β): {achieved_power:.2%}") print(f"Type II Error Risk (β): {1 - achieved_power:.2%}") if achieved_power >= 0.80: print("✅ Power requirement met (≥ 80%). Low risk of false negatives.") else: print("⚠️ Under-powered test! Increase sample size to avoid Type II errors.") ``` ## Statistical power - the missing concept **Power** = 1 − β = the probability of correctly detecting a real effect. Common convention is 80% power. That means even when a real effect exists, you'll fail to detect it 20% of the time. ## Takeaway There is no universally correct α or β. The "right" choice depends on which error costs more in your specific business context.