Case 3 — Run an A/B Test Between Two Model Versions

Enterprise Scenario: Your team has developed a new churn prediction challenger model (v1.3.0) designed to replace incumbent champion v1.2.0. Offline evaluation suggests v1.3.0 improves PR-AUC by 0.025. However, deploying the model across all 200,000 active subscribers immediately introduces revenue risk. You must design and simulate a rigorous A/B experiment: implement deterministic hash routing to allocate users across variants, run statistical assertions to detect Sample Ratio Mismatch (SRM), track guardrail metrics, and formulate an evidence-based promotion decision.

Technical Requirements & Deliverables

Deliverable 1: Experiment Design Specification

• Define the formal hypothesis ($H_0$: $ ext{Lift} \le 0$, $H_1$: $ ext{Lift} > 0$).
• Design the traffic allocation: 90% Champion (v1.2.0), 5% Challenger (v1.3.0), 5% Baseline Holdout (v1.0.0).
• Define Primary Metric (Precision-Recall AUC on confirmed retention outcomes) and Guardrail Metrics (Prediction p99 latency < 60ms, false positive retention intervention rate < 5%).

Deliverable 2: Deterministic Hash Router (routing.py)

• Implement a hash routing function: assign_variant(user_id: str, salt: str = "exp_churn_2026") -> str.
• Use MD5/SHA256 hashing modulo 100 to guarantee that a customer is 100% consistently assigned to the same variant across repeated visits without requiring stateful database lookups.

Deliverable 3: Experiment Telemetry & Statistical Testing (evaluate_experiment.py)

• Ingest simulated user interaction logs across the three cohorts.
• Perform a Chi-square goodness-of-fit test to assert that traffic matches the 90/5/5 ratio; flag Sample Ratio Mismatch (SRM) if $p < 0.01$.
• Compute two-proportion z-tests and 95% confidence intervals on retention intervention conversion rates.
• Evaluate latency p95 and p99 across champion and challenger variants.

Deliverable 4: Promotion & Rollback Runbook

• Formulate explicit quantitative criteria for: (a) 100% promotion to Champion, (b) extending experiment duration, or (c) immediate abort and rollback.

Starter Code Scaffold

import hashlib
import numpy as np
from scipy import stats

def assign_variant(user_id: str, salt: str = "exp_churn_2026") -> str:
    # Deterministically map user_id to variant bucket using salted hash modulo 100.
    raw_hash = hashlib.md5(f"{user_id}_{salt}".encode("utf-8")).hexdigest()
    bucket = int(raw_hash, 16) % 100
    
    if bucket < 90:
        return "champion_v1.2"
    elif bucket < 95:
        return "challenger_v1.3"
    else:
        return "holdout_v1.0"

def check_sample_ratio_mismatch(observed_counts: list[int], expected_proportions: list[float]) -> dict:
    # Compute Chi-square goodness-of-fit test for traffic allocation.
    total = sum(observed_counts)
    expected_counts = [total * p for p in expected_proportions]
    chi2_stat, p_value = stats.chisquare(f_obs=observed_counts, f_exp=expected_counts)
    return {
        "chi2_stat": float(chi2_stat),
        "p_value": float(p_value),
        "srm_detected": bool(p_value < 0.01)
    }

def evaluate_two_proportion_lift(n_ctrl: int, conv_ctrl: int, n_treat: int, conv_treat: int) -> dict:
    # Compute lift, z-score, p-value, and 95% confidence intervals.
    # TODO: Implement two-proportion z-test calculation
    pass

100-Point Graded Rubric

• Criteria 1: Experiment Design & Metric Formulation (25 pts)
  • 25 pts: Explicitly pre-registers hypothesis, statistical power calculation, MDE, primary metric, and latency guardrails.
  • 15 pts: Basic metric definitions provided but lacks power calculation or latency guardrail specifications.
  • 0 pts: No formal hypothesis or guardrail criteria defined.
• Criteria 2: Deterministic Hash Routing (25 pts)
  • 25 pts: Implements salted MD5/SHA256 hashing modulo 100; guarantees 100% assignment consistency for identical user IDs.
  • 15 pts: Hashing implemented but lacks salt configuration or handles edge distributions incorrectly.
  • 0 pts: Uses random non-deterministic sampling (random.random()).
• Criteria 3: SRM Diagnostics & Telemetry (25 pts)
  • 25 pts: Implements Chi-square SRM validation test; accurately flags allocation bias before business metric calculation.
  • 15 pts: Computes metric differences but skips Chi-square SRM allocation verification.
  • 0 pts: Fails to validate traffic distribution against expected ratios.
• Criteria 4: Statistical Decision Protocol & Runbook (25 pts)
  • 25 pts: Computes confidence intervals on lift; establishes unambiguous thresholds for promotion vs. rollback.
  • 15 pts: Calculates p-values but lacks confidence interval boundaries or concrete rollback protocols.
  • 0 pts: Recommends promotion based on raw metric lift without statistical significance.