Your Analyst Checklists - Reusable Mental Models

These are the five mental checklists you'll use most as a working analyst.

Checklist 1 - Before I Trust This Number

• Source / pipeline / query verified?
• Time window consistent?
• Unique users vs events?

Checklist 2 - Before I Compute "Average X"

• Distribution shape checked?
• Skew ratio checked?
• Percentiles reported?

Checklist 3 - Before I Trust an A/B Test Result

• SRM chi-square test passed?
• Novelty fade checked?
• Guardrails held?

Automated Experiment Audit Suite in Python

import numpy as np
from scipy import stats
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep

def automated_experiment_audit(n_c, conv_c, n_t, conv_t, mde=0.005):
    # Executes the complete analyst checklist in code
    print("================ EXPERIMENT AUDIT ================")
    # Check 1: SRM
    chi2, srm_p = stats.chisquare([n_c, n_t], [(n_c+n_t)/2, (n_c+n_t)/2])
    print(f"1. SRM Check: Chi2={chi2:.2f}, p={srm_p:.4f} -> {'PASS' if srm_p > 0.01 else 'FAIL (SRM)'}")

    # Check 2: Statistical Significance & CI
    p_c, p_t = conv_c / n_c, conv_t / n_t
    lift = p_t - p_c
    z, p_val = proportions_ztest([conv_t, conv_c], [n_t, n_c])
    ci_low, ci_high = confint_proportions_2indep(conv_t, n_t, conv_c, n_c)

    print(f"2. Primary Lift: {lift:+.2%} (95% CI: [{ci_low:+.2%}, {ci_high:+.2%}], p={p_val:.4f})")

    # Check 3: MDE Ship Barrier
    cleared_mde = lift >= mde and ci_low > 0
    print(f"3. MDE Barrier (≥ {mde:.2%}): {'PASS' if cleared_mde else 'FAIL'}")

    # Final recommendation
    if srm_p > 0.01 and cleared_mde:
        print("RECOMMENDATION: 🚀 SHIP TREATMENT")
    else:
        print("RECOMMENDATION: 🛑 DO NOT SHIP / INVESTIGATE")
    print("==================================================")

automated_experiment_audit(n_c=25000, conv_c=2100, n_t=25000, conv_t=2350, mde=0.005)

Takeaway

Mastering checklists turns stats theory into everyday workplace excellence.