The Test Design Checklist

# The Test Design Checklist Most A/B test problems trace back to bad design, not bad analysis. ## Test specification dataclass in Python ```python from dataclasses import dataclass from typing import List @dataclass class ABTestSpecification: test_name: str hypothesis: str primary_metric: str baseline_rate: float mde_target: float guardrail_metrics: List[str] exposure_filter: str planned_duration_days: int planned_sample_per_arm: int spec = ABTestSpecification( test_name="checkout_simplified_v2", hypothesis="Removing coupon field on step 1 increases checkout completion by ≥1.5pp without reducing AOV", primary_metric="checkout_completion_rate", baseline_rate=0.082, mde_target=0.015, guardrail_metrics=["average_order_value_inr", "refund_rate_30d"], exposure_filter="all_logged_in_web_users", planned_duration_days=14, planned_sample_per_arm=12500 ) print(f"Experiment: {spec.test_name}") print(f"Target Primary MDE: +{spec.mde_target:.1%}") print(f"Guardrails: {', '.join(spec.guardrail_metrics)}") ``` ## Takeaway Lock the primary metric, guardrails, and decision rules in a written pre-flight specification before turning on experiment traffic.