Distributions to Hypothesis Tests Pointer

This lesson on Distributions to Hypothesis Tests Pointer is hands-on and example-driven. You will quickly diagnose experimental setups and map business questions directly to the correct statistical hypothesis tests. By evaluating data types, sample sizes, and distribution assumptions, you will select appropriate parametric or non-parametric tests for continuous and categorical metrics.

What You'll Be Able To Do

• Identify whether a metric requires parametric or non-parametric statistical evaluation based on distribution normality.
• Select between independent t-tests, paired t-tests, and Mann-Whitney U tests for two-sample comparisons.
• Apply one-way ANOVA or Kruskal-Wallis tests when comparing continuous outcomes across three or more variants.
• Determine when to use Chi-Square tests of independence versus two-proportion z-tests for categorical outcomes.

Detailed Concept Walkthrough

1. Test Selection Decision Framework

Choosing the appropriate hypothesis test depends systematically on the number of groups, the scale of measurement, and distribution normality. You treat statistical testing as a deterministic decision tree starting from metric type down to parametric assumptions.

• Metric Classification: You first determine whether the target metric is continuous (e.g., revenue, session duration) or categorical/discrete (e.g., conversion rate, click-through). This branch splits your options between mean comparisons and proportion/frequency tests.
• Group Count and Independence: You count the distinct comparison groups (one sample, two samples, or K samples) and verify whether the samples are independent or paired across time and subjects.
• Assumption Validation: You evaluate whether continuous metrics satisfy normality via visual inspection or normality tests like Shapiro-Wilk, defaulting to non-parametric tests when sample sizes are small and skewed.

from scipy import stats

# Step 1: Test normality of continuous sample data
stat, p_val = stats.shapiro(sample_data)
is_normal = p_val > 0.05
print(f"Normal distribution: {is_normal} (p={p_val:.4f})")

Key Takeaway: Map the outcome metric type, group count, and normality profile before executing any hypothesis test.

2. Two-Sample Continuous Comparisons

Two-sample tests evaluate whether the central tendencies of two distinct groups differ significantly. You use parametric Student's or Welch's t-tests for normal distributions and rank-based tests when normality fails.

• Parametric Two-Sample t-test: You compute Welch's t-test when comparing continuous means between two independent groups without assuming equal variance across samples. The test calculates the t-statistic representing standard errors between group means.
• Non-Parametric Mann-Whitney U: When data is heavily skewed or ordinal, you convert continuous values to ranks to test for stochastic dominance rather than mean equality.
• Paired Comparisons: For before-and-after measurements on identical subjects, you apply paired t-tests or Wilcoxon signed-rank tests to isolate within-subject variance.

from scipy import stats

# Welch's t-test (parametric, independent groups)
t_stat, p_val_t = stats.ttest_ind(group_a, group_b, equal_var=False)

# Mann-Whitney U test (non-parametric, independent groups)
u_stat, p_val_u = stats.mannwhitneyu(group_a, group_b, alternative='two-sided')

Key Takeaway: Default to Welch's t-test for continuous means and Mann-Whitney U when normality assumptions are violated.

3. Multi-Group Continuous Comparisons

When comparing continuous metrics across three or more variants, multi-group tests control the family-wise error rate rather than running repeated pairwise tests. You assess total variance partitioned into between-group and within-group components.

• One-Way ANOVA: You compute the F-statistic to compare the variance between group means against the variance within each group under normality assumptions.
• Non-Parametric Kruskal-Wallis: When multi-group continuous data is non-normal, Kruskal-Wallis ranks all combined observations to evaluate whether median ranks differ across groups.
• Post-Hoc Adjustments: When the omnibus test rejects the null, you run Tukey's HSD or Bonferroni-corrected pairwise tests to determine which specific groups diverge.

from scipy import stats

# One-Way ANOVA across 3 test variants
f_stat, p_val_anova = stats.f_oneway(control, variant_1, variant_2)

# Kruskal-Wallis non-parametric alternative
h_stat, p_val_kw = stats.kruskal(control, variant_1, variant_2)

Key Takeaway: Use ANOVA or Kruskal-Wallis to prevent alpha inflation before conducting targeted pairwise post-hoc tests.

4. Categorical and Frequency Tests

Categorical hypothesis tests determine whether observed frequencies or proportions diverge from expected baseline distributions. You evaluate contingency tables to test independence or differences in conversion rates.

• Chi-Square Test of Independence: You build an M x N contingency table to test whether two categorical variables are independent based on observed versus expected cell counts.
• Two-Proportion Z-Test: For binary outcomes across two groups (e.g., converted vs dropped), you calculate standard normal deviations between pooled proportions.
• Cell Count Constraints: You ensure every contingency table cell has an expected count of at least 5; otherwise, you fall back to Fisher's Exact Test.

import numpy as np
from scipy import stats

# Contingency table: [[converted_A, bounced_A], [converted_B, bounced_B]]
contingency_table = np.array([[120, 880], [150, 850]])
chi2, p_val, dof, expected = stats.chi2_contingency(contingency_table)

Key Takeaway: Analyze categorical proportions using Chi-Square contingency tests while validating minimum expected cell frequencies.

Topics Covered in Distributions to Hypothesis Tests Pointer

1. Decision Framework Overview (0:00 - 1:30) — The test selection decision tree based on metric types and group configurations is introduced.
2. Two-Sample Continuous Testing (1:30 - 3:30) — Selection criteria between Student's t-test, Welch's t-test, and Mann-Whitney U are compared.
3. Multi-Group Continuous Evaluation (3:30 - 5:30) — ANOVA and Kruskal-Wallis implementations are detailed to manage multiple treatment variants.
4. Categorical Metric Analysis (5:30 - 7:00) — Chi-Square contingency tests and proportion testing mechanics are walked through.
5. Scenario Practice Summary (7:00 - 8:00) — Three practical business scenarios are resolved using the test selection framework.

DS Bridge Cheat Sheet

• stats.shapiro(data) — Tests sample normality using Shapiro-Wilk test

  stat, p = stats.shapiro(df['revenue'])
  

• stats.ttest_ind(a, b, equal_var=False) — Runs Welch independent two-sample t-test

  stat, p = stats.ttest_ind(ctrl, exp, equal_var=False)
  

• stats.mannwhitneyu(a, b) — Runs non-parametric rank test for two groups

  stat, p = stats.mannwhitneyu(ctrl, exp)
  

• stats.f_oneway(*groups) — Computes one-way ANOVA across multiple groups

  stat, p = stats.f_oneway(g1, g2, g3)
  

• stats.kruskal(*groups) — Computes non-parametric ANOVA on ranked data

  stat, p = stats.kruskal(g1, g2, g3)
  

• stats.chi2_contingency(table) — Tests independence in categorical contingency table

  chi2, p, _, _ = stats.chi2_contingency(table)
  

Comparison Table

Common Pitfalls

• Mistake: Running multiple pairwise t-tests across three variants. Avoid: Run ANOVA or Kruskal-Wallis first to control alpha inflation.
• Mistake: Assuming equal variances when running independent two-sample t-tests. Avoid: Explicitly set equal_var=False to use Welch t-test.
• Mistake: Applying Chi-Square tests to small datasets with expected counts under 5. Avoid: Use Fisher Exact Test for sparse contingency tables.

FAQs

• When should I choose Welch's t-test over a standard Student's t-test? Always default to Welch's t-test unless you have verified equal variances between groups. Welch's maintains robust Type I error control without sacrificing statistical power.
• What test should I run for skewed continuous metrics like user spend? Use the Mann-Whitney U test for two groups or Kruskal-Wallis for multiple groups. Alternatively, log-transform the continuous metric if normality can be recovered.
• How does a paired t-test differ from an independent t-test? A paired t-test evaluates differences within the same subjects measured across two conditions, whereas an independent t-test compares distinct, unrelated subject groups.