SUBMISSION: Project 3 — Bootstrap Confidence Intervals & Permutation Tests on Non-Normal Retention Metrics

🎯 Hands-on Portfolio Deliverable. Non-normal metrics are the norm in product data. This project proves you can do inference without hiding behind normality.

🏢 Business Scenario & Executive Prompt

FitTrack (fitness app, 5,000 paid cohorts) reports "median 30-day retention is 41% — up from 38% after the streaks feature." The CFO asks two questions: (1) "What is the 95% CI on that median — the distribution is extremely skewed, so your usual formula does not apply," and (2) "Is the 3-point lift real, or could random assignment have produced it?" Retention days are zero-inflated and heavy-tailed: the t-interval is invalid here.

📋 Source Data & Input Schemas

fittrack_retention.csv — 5,000 rows (mock shape; scaffold generates it):

🎯 Required Deliverables

1. bootstrap_permutation.ipynb — EDA → bootstrap CI → permutation test → failure-mode demo.
2. retention_inference.csv — statistic, estimate, ci_lo, ci_hi, method, n_boot (median overall, median by cohort, difference-in-medians).
3. retention_memo.md — CI for the median, permutation p-value verdict on the lift, and one paragraph on when you would NOT bootstrap.
4. bootstrap_dist.png — bootstrap distribution of the median with CI bands marked.

🛠️ Step-by-Step Implementation Guide

1. Show the t-interval failing. Compute mean ± 1.96·SE on retention_days_30; overlay a normal curve on the skewed histogram to show the mismatch. Explain why the median is the right summary.
2. Percentile bootstrap CI. 10,000 resamples (seeded) of the median; 2.5/97.5 percentiles → 95% CI. Plot the bootstrap distribution with bands.
3. Permutation test for the lift. Shuffle cohort labels 10,000×, build the null distribution of difference-in-medians, exact p-value = fraction ≥ observed. Contrast with a naive t-test p-value.
4. Break it on purpose. Subsample n=15 tail quantiles (99th percentile) and show bootstrap instability; run an AR(1)-correlated series through naive bootstrap vs block bootstrap to demo dependence failure.
5. Memo. Numbers + verdict + limits (tails, dependence, small-n).

💻 Production Starter Scaffold

import numpy as np, pandas as pd
rng = np.random.default_rng(11)

def retention_mix(n, zero_p=0.35):
    zeros = rng.binomial(1, zero_p, n)
    vals = rng.negative_binomial(n=3, p=0.25, size=n).clip(0,30)
    return np.where(zeros, 0, vals)

df = pd.DataFrame({
    "user_id": [f"FT-{i:05d}" for i in range(5000)],
    "cohort": ["control"]*2500 + ["streaks"]*2500,
    "retention_days_30": np.concatenate([retention_mix(2500), retention_mix(2500)+rng.binomial(1,.12,2500)]),
    "plan": rng.choice(["monthly","annual"],5000,p=[.7,.3]),
    "acq_channel": rng.choice(["organic","paid","referral"],5000,p=[.5,.3,.2]),
})

def bootstrap_ci(x, stat=np.median, n_boot=10_000, alpha=0.05, seed=11):
    r = np.random.default_rng(seed)
    boots = np.array([stat(r.choice(x, size=len(x), replace=True)) for _ in range(n_boot)])
    return boots, tuple(np.percentile(boots, [100*alpha/2, 100*(1-alpha/2)]))

def permutation_p(a, b, n_perm=10_000, seed=11):
    r = np.random.default_rng(seed)
    obs = np.median(b) - np.median(a)
    pool = np.concatenate([a,b]); na = len(a)
    null = np.array([np.median(r.permutation(pool)[na:]) - np.median(r.permutation(pool)[:na]) for _ in range(n_perm)])
    return obs, float((np.abs(null) >= abs(obs)).mean())

med = df.retention_days_30.to_numpy()
boots, ci = bootstrap_ci(med)
print("median CI:", np.median(med), ci)
a = df.query("cohort=='control'").retention_days_30.to_numpy()
b = df.query("cohort=='streaks'").retention_days_30.to_numpy()
print("lift, perm p:", permutation_p(a, b))

🧪 Acceptance Criteria & Verification Checks

• t-interval-vs-histogram overlay visibly demonstrates the normality mismatch.
• Median 95% CI from 10,000 seeded bootstraps; bootstrap_dist.png shows bands at 2.5/97.5%.
• Permutation p-value computed from a 10,000-shuffle null (not a t-test) and reported with the observed lift.
• Failure demo: tail-quantile instability + block-vs-naive bootstrap on correlated series, both shown.
• Memo answers all three executive questions with numbers.

📊 Evaluation Rubric (100 points — passing ≥ 80)

📤 Submission Guidelines

1. Upload your files (.ipynb, .csv, .md, .png) via the file dropzone on this lesson.
2. Paste your public GitHub repo URL in the submission notes (repo must contain the same files + a 5-line README with the run command).
3. Name files exactly as listed under Required Deliverables — autograding checks filenames first.
4. Passing score is 80/100. You may resubmit; the highest score counts.