Reading A/B Test Results

This lesson on Reading A/B Test Results is hands-on and example-driven. You will master the end-to-end framework for designing, executing, and interpreting A/B tests in product analytics. You will be able to select robust invariant and evaluation metrics, choose the proper unit of diversion, calculate sample size requirements, and evaluate statistical significance versus business trade-offs.

What You'll Be Able To Do

• Differentiate invariant metrics from evaluation metrics to detect experimental bias and measure treatment effects.
• Select the appropriate unit of diversion based on user authentication state and user experience consistency.
• Formulate null and alternative hypotheses to evaluate conversion rate shifts.
• Determine required sample size using power, significance level, and minimum detectable effect.
• Interpret p-values and confidence intervals to make launch decisions while accounting for business metric trade-offs.

Detailed Concept Walkthrough

1. Unit of Diversion and User Randomization

The unit of diversion defines the entity level at which users or sessions are randomly assigned to either the control or treatment group. Choosing the right unit prevents user experience contamination and maintains data integrity.

• Mechanism: Common units include cookie-based IDs, signed-in user IDs, and session IDs. Signed-in user IDs provide cross-device consistency, whereas cookie-based diversion works best for unauthenticated web traffic.
• Execution Flow: A deterministic hash (e.g., MD5 or MurmurHash) mod 100 maps the chosen identifier to a bucket between 0 and 99, assigning the user deterministically to Control (0-49) or Treatment (50-99).
• Best Practice: Choose a unit of diversion that aligns directly with the unit of analysis; mismatched units (e.g., randomizing by session while measuring user-level retention) violate independence assumptions.

# Deterministic user bucketing using hash modulo
import hashlib

def get_variant(user_id: str, salt: str = 'exp_v1') -> str:
    raw = f'{user_id}:{salt}'.encode('utf-8')
    bucket = int(hashlib.md5(raw).hexdigest(), 16) % 100
    return 'treatment' if bucket >= 50 else 'control'

print(get_variant('usr_10492'))  # Deterministic output

Key Takeaway: Always match the randomization unit to the user journey stage to prevent variant spillover and split-brain user experiences.

2. Metric Selection: Invariant vs Evaluation Metrics

A rigorous experiment requires two distinct classes of metrics: invariant metrics to verify experiment validity and evaluation metrics to measure business impact.

• Mechanism: Invariant metrics (e.g., total traffic volume, device split, demographic distribution) must remain statistically equal across control and treatment groups to rule out sample ratio mismatch (SRM).
• Under the Hood: Evaluation metrics (e.g., checkout conversion rate, average revenue per user) reflect the direct causal effect of the product change defined in the alternative hypothesis.
• Best Practice: Check invariant metrics before analyzing evaluation metrics; if invariant metrics fail their hypothesis test, the evaluation metrics cannot be trusted.

# Verify invariant metric (Sample Ratio Mismatch check via Chi-Square)
from scipy.stats import chisquare

observed_traffic = [50200, 49800]  # [control, treatment]
expected_traffic = [50000, 50000]  # 50/50 split target

chi2_stat, p_val = chisquare(f_obs=observed_traffic, f_exp=expected_traffic)
srm_detected = p_val < 0.01  # Invariant metric holds if p_val >= 0.01

Key Takeaway: Invariant metrics ensure the experimental setup is unbiased before evaluation metrics are interpreted for business impact.

3. Power Analysis and Sample Size Sizing

Power analysis establishes the minimum sample size needed to detect a predetermined minimum detectable effect (MDE) with high statistical confidence before launching the test.

• Mechanism: Sample size calculation balances significance level (alpha, typically 0.05 / Type I error) against statistical power (1 - beta, typically 0.80 / Type II error prevention).
• Execution Flow: Smaller MDEs require quadratically larger sample sizes; detecting a 1% lift demands substantially more traffic and run time than detecting a 10% lift.
• Best Practice: Fix the test duration to full-week multiples (e.g., 14 or 28 days) to capture weekly seasonal cycles and prevent day-of-week distortion.

# Sample size calculation for two-proportion comparison
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

base_cr, target_cr = 0.10, 0.11  # 10% baseline, 1% absolute lift (MDE)
effect_size = proportion_effectsize(base_cr, target_cr)
power_analysis = NormalIndPower()

required_n_per_variant = power_analysis.solve_power(
    effect_size=effect_size, alpha=0.05, power=0.80, ratio=1.0
)
print(f'Required N per group: {int(required_n_per_variant):,}')

Key Takeaway: Determine sample size prior to test deployment to guarantee adequate statistical power and avoid underpowered false negatives.

4. Hypothesis Testing and Statistical Significance

Hypothesis testing uses p-values and confidence intervals to determine whether observed differences in evaluation metrics reflect true causal effects or random noise.

• Mechanism: The Null Hypothesis ($H_0$) assumes zero difference between variants, while the Alternative Hypothesis ($H_1$) asserts a real treatment effect.
• Under the Hood: A p-value below alpha (e.g., p < 0.05) rejects $H_0$, indicating that the observed difference has less than a 5% probability of occurring under the null model.
• Best Practice: Rely on confidence intervals rather than p-values alone to understand both the precision of the lift estimate and the practical business boundaries of the effect.

# Evaluate conversion lift with Z-Test and 95% Confidence Interval
import numpy as np
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep

successes = np.array([1000, 1150])  # [control, treatment]
nobs = np.array([10000, 10000])

z_stat, p_val = proportions_ztest(count=successes, nobs=nobs)
ci_low, ci_high = confint_proportions_2indep(successes[1], nobs[1], successes[0], nobs[0], method='wald')

print(f'P-Value: {p_val:.4f} | 95% CI of Lift: [{ci_low:.4f}, {ci_high:.4f}]')

Key Takeaway: Reject the null hypothesis only when the p-value meets the alpha threshold, and use confidence intervals to assess business risk.

Topics Covered in Reading A/B Test Results

1. Experiment Overview (0:00 - 2:02) — Introduces the end-to-end A/B testing workflow and foundational concepts for analytics interviews.
2. Unit of Diversion (2:02 - 5:50) — Explains user-level versus cookie-level randomization strategies and their operational trade-offs.
3. Metric Framework (5:50 - 6:28) — Defines the roles of invariant sanity check metrics and business evaluation metrics.
4. Power and Sizing (6:28 - 11:08) — Outlines sample size estimation principles involving statistical power, alpha thresholds, and sensitivity.
5. Hypothesis Interpretation (11:08 - 13:39) — Demonstrates interpreting p-values, confidence intervals, and business trade-offs for launch decisions.

Stats & Product Analytics for Analysts Cheat Sheet

• Unit of Diversion — Defines randomization entity level across variants

  unit_type = 'user_id' if is_authenticated else 'cookie_id'
  

• Invariant Metric — Checks balance and integrity across groups

  assert abs(control_users - treat_users) / total_users < 0.01
  

• Evaluation Metric — Measures core business optimization objective

  eval_metric = converted_purchases / total_visitors
  

• proportions_ztest() — Calculates z-score and p-value for proportions

  z_stat, p_val = proportions_ztest([120, 150], [1000, 1000])
  

• proportion_effectsize() — Computes standardized effect size for proportions

  es = proportion_effectsize(prop1=0.05, prop2=0.055)
  

• solve_power() — Computes required sample size per variant

  NormalIndPower().solve_power(effect_size=es, alpha=0.05, power=0.8)
  

Comparison Table

Common Pitfalls

• Mistake: Checking evaluation metrics while invariant metrics show imbalance. Avoid: Always validate sample ratio consistency across control and treatment before analyzing success metrics.
• Mistake: Peeking at results and stopping early upon reaching nominal significance. Avoid: Precompute required sample size with power analysis and run tests to full completion.
• Mistake: Using session ID diversion for features requiring persistent user experience. Avoid: Use persistent user or cookie IDs when user familiarity affects behavior.

FAQs

• What should I do if an invariant metric shows a statistically significant difference? Do not interpret evaluation metrics; investigate upstream bugs, such as tracking failures, redirection delays, or broken randomization logic.
• How do statistical significance and practical significance differ? Statistical significance confirms an effect is not random noise, whereas practical significance checks whether the lift justifies engineering, maintenance, and opportunity costs.
• Why is cookie-based diversion risky for cross-device web products? A single user switching between phone and desktop gets assigned to different buckets, contaminating treatment exposure and degrading measurement accuracy.