Most A/B test problems trace back to bad design, not bad analysis.
Test specification dataclass in Python
from dataclasses import dataclass
from typing import List
@dataclass
class ABTestSpecification:
test_name: str
hypothesis: str
primary_metric: str
baseline_rate: float
mde_target: float
guardrail_metrics: List[str]
exposure_filter: str
planned_duration_days: int
planned_sample_per_arm: int
spec = ABTestSpecification(
test_name="checkout_simplified_v2",
hypothesis="Removing coupon field on step 1 increases checkout completion by ≥1.5pp without reducing AOV",
primary_metric="checkout_completion_rate",
baseline_rate=0.082,
mde_target=0.015,
guardrail_metrics=["average_order_value_inr", "refund_rate_30d"],
exposure_filter="all_logged_in_web_users",
planned_duration_days=14,
planned_sample_per_arm=12500
)
print(f"Experiment: {spec.test_name}")
print(f"Target Primary MDE: +{spec.mde_target:.1%}")
print(f"Guardrails: {', '.join(spec.guardrail_metrics)}")
Takeaway
Lock the primary metric, guardrails, and decision rules in a written pre-flight specification before turning on experiment traffic.