Your Analyst Checklists - Reusable Mental Models

# Your Analyst Checklists These are the five mental checklists you'll use most as a working analyst. ## Checklist 1 - Before I Trust This Number - Source / pipeline / query verified? - Time window consistent? - Unique users vs events? ## Checklist 2 - Before I Compute "Average X" - Distribution shape checked? - Skew ratio checked? - Percentiles reported? ## Checklist 3 - Before I Trust an A/B Test Result - SRM chi-square test passed? - Novelty fade checked? - Guardrails held? ## Automated Experiment Audit Suite in Python ```python import numpy as np from scipy import stats from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep def automated_experiment_audit(n_c, conv_c, n_t, conv_t, mde=0.005): # Executes the complete analyst checklist in code print("================ EXPERIMENT AUDIT ================") # Check 1: SRM chi2, srm_p = stats.chisquare([n_c, n_t], [(n_c+n_t)/2, (n_c+n_t)/2]) print(f"1. SRM Check: Chi2={chi2:.2f}, p={srm_p:.4f} -> {'PASS' if srm_p > 0.01 else 'FAIL (SRM)'}") # Check 2: Statistical Significance & CI p_c, p_t = conv_c / n_c, conv_t / n_t lift = p_t - p_c z, p_val = proportions_ztest([conv_t, conv_c], [n_t, n_c]) ci_low, ci_high = confint_proportions_2indep(conv_t, n_t, conv_c, n_c) print(f"2. Primary Lift: {lift:+.2%} (95% CI: [{ci_low:+.2%}, {ci_high:+.2%}], p={p_val:.4f})") # Check 3: MDE Ship Barrier cleared_mde = lift >= mde and ci_low > 0 print(f"3. MDE Barrier (≥ {mde:.2%}): {'PASS' if cleared_mde else 'FAIL'}") # Final recommendation if srm_p > 0.01 and cleared_mde: print("RECOMMENDATION: 🚀 SHIP TREATMENT") else: print("RECOMMENDATION: 🛑 DO NOT SHIP / INVESTIGATE") print("==================================================") automated_experiment_audit(n_c=25000, conv_c=2100, n_t=25000, conv_t=2350, mde=0.005) ``` ## Takeaway Mastering checklists turns stats theory into everyday workplace excellence.