Which Error Costs You More?

Two errors are possible in any hypothesis test:

• Type I error (false positive) - rejecting the null when it's actually true. You ship a treatment that doesn't actually work.
• Type II error (false negative) - failing to reject the null when the alternative is actually true. You DON'T ship a treatment that does work.

Calculating Statistical Power in Python

import numpy as np
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

# Baseline conversion = 10% (0.10), Expected Treatment = 11.5% (0.115) -> MDE = 1.5pp
p1 = 0.10
p2 = 0.115
effect_size = proportion_effectsize(p1, p2)

power_analyzer = NormalIndPower()

# Calculate power for N = 5,000 per arm at alpha = 0.05
n_sample = 5000
alpha = 0.05
achieved_power = power_analyzer.solve_power(
    effect_size=effect_size,
    nobs1=n_sample,
    alpha=alpha,
    power=None,
    ratio=1.0
)

print(f"Effect size (Cohen's h): {effect_size:.4f}")
print(f"Sample per arm:          {n_sample:,}")
print(f"Statistical Power (1-β): {achieved_power:.2%}")
print(f"Type II Error Risk (β):  {1 - achieved_power:.2%}")

if achieved_power >= 0.80:
    print("✅ Power requirement met (≥ 80%). Low risk of false negatives.")
else:
    print("⚠️ Under-powered test! Increase sample size to avoid Type II errors.")

Statistical power - the missing concept

Power = 1 − β = the probability of correctly detecting a real effect.

Common convention is 80% power. That means even when a real effect exists, you'll fail to detect it 20% of the time.

Takeaway

There is no universally correct α or β. The "right" choice depends on which error costs more in your specific business context.