Case 3 — Run an A/B Test Between Two Model Versions

# Case 3 — Run an A/B Test Between Two Model Versions > **Enterprise Scenario:** Your team has developed a new churn prediction challenger model (`v1.3.0`) designed to replace incumbent champion `v1.2.0`. Offline evaluation suggests `v1.3.0` improves PR-AUC by 0.025. However, deploying the model across all 200,000 active subscribers immediately introduces revenue risk. You must design and simulate a rigorous A/B experiment: implement deterministic hash routing to allocate users across variants, run statistical assertions to detect Sample Ratio Mismatch (SRM), track guardrail metrics, and formulate an evidence-based promotion decision. --- ## Technical Requirements & Deliverables ### Deliverable 1: Experiment Design Specification - Define the formal hypothesis ($H_0$: $ ext{Lift} \le 0$, $H_1$: $ ext{Lift} > 0$). - Design the traffic allocation: 90% Champion (`v1.2.0`), 5% Challenger (`v1.3.0`), 5% Baseline Holdout (`v1.0.0`). - Define Primary Metric (Precision-Recall AUC on confirmed retention outcomes) and Guardrail Metrics (Prediction p99 latency str`. - Use MD5/SHA256 hashing modulo 100 to guarantee that a customer is 100% consistently assigned to the same variant across repeated visits without requiring stateful database lookups. ### Deliverable 3: Experiment Telemetry & Statistical Testing (`evaluate_experiment.py`) - Ingest simulated user interaction logs across the three cohorts. - Perform a Chi-square goodness-of-fit test to assert that traffic matches the 90/5/5 ratio; flag Sample Ratio Mismatch (SRM) if $p < 0.01$. - Compute two-proportion z-tests and 95% confidence intervals on retention intervention conversion rates. - Evaluate latency p95 and p99 across champion and challenger variants. ### Deliverable 4: Promotion & Rollback Runbook - Formulate explicit quantitative criteria for: (a) 100% promotion to Champion, (b) extending experiment duration, or (c) immediate abort and rollback. --- ## Starter Code Scaffold ```python import hashlib import numpy as np from scipy import stats def assign_variant(user_id: str, salt: str = "exp_churn_2026") -> str: # Deterministically map user_id to variant bucket using salted hash modulo 100. raw_hash = hashlib.md5(f"{user_id}_{salt}".encode("utf-8")).hexdigest() bucket = int(raw_hash, 16) % 100 if bucket dict: # Compute Chi-square goodness-of-fit test for traffic allocation. total = sum(observed_counts) expected_counts = [total * p for p in expected_proportions] chi2_stat, p_value = stats.chisquare(f_obs=observed_counts, f_exp=expected_counts) return { "chi2_stat": float(chi2_stat), "p_value": float(p_value), "srm_detected": bool(p_value dict: # Compute lift, z-score, p-value, and 95% confidence intervals. # TODO: Implement two-proportion z-test calculation pass ``` --- ## 100-Point Graded Rubric * **Criteria 1: Experiment Design & Metric Formulation (25 pts)** - *25 pts:* Explicitly pre-registers hypothesis, statistical power calculation, MDE, primary metric, and latency guardrails. - *15 pts:* Basic metric definitions provided but lacks power calculation or latency guardrail specifications. - *0 pts:* No formal hypothesis or guardrail criteria defined. * **Criteria 2: Deterministic Hash Routing (25 pts)** - *25 pts:* Implements salted MD5/SHA256 hashing modulo 100; guarantees 100% assignment consistency for identical user IDs. - *15 pts:* Hashing implemented but lacks salt configuration or handles edge distributions incorrectly. - *0 pts:* Uses random non-deterministic sampling (`random.random()`). * **Criteria 3: SRM Diagnostics & Telemetry (25 pts)** - *25 pts:* Implements Chi-square SRM validation test; accurately flags allocation bias before business metric calculation. - *15 pts:* Computes metric differences but skips Chi-square SRM allocation verification. - *0 pts:* Fails to validate traffic distribution against expected ratios. * **Criteria 4: Statistical Decision Protocol & Runbook (25 pts)** - *25 pts:* Computes confidence intervals on lift; establishes unambiguous thresholds for promotion vs. rollback. - *15 pts:* Calculates p-values but lacks confidence interval boundaries or concrete rollback protocols. - *0 pts:* Recommends promotion based on raw metric lift without statistical significance.