SUBMISSION: Project 3 — Bootstrap Confidence Intervals & Permutation Tests on Non-Normal Retention Metrics

# SUBMISSION: Project 3 — Bootstrap Confidence Intervals & Permutation Tests on Non-Normal Retention Metrics > 🎯 **Hands-on Portfolio Deliverable.** Non-normal metrics are the norm in product data. This project proves you can do inference *without* hiding behind normality. ## 🏢 Business Scenario & Executive Prompt **FitTrack** (fitness app, 5,000 paid cohorts) reports *"median 30-day retention is 41% — up from 38% after the streaks feature."* The CFO asks two questions: (1) *"What is the 95% CI on that median — the distribution is extremely skewed, so your usual formula does not apply,"* and (2) *"Is the 3-point lift real, or could random assignment have produced it?"* Retention days are zero-inflated and heavy-tailed: the t-interval is invalid here. ## 📋 Source Data & Input Schemas **`fittrack_retention.csv`** — 5,000 rows (mock shape; scaffold generates it): | column | dtype | description | |---|---|---| | `user_id` | str | unique user | | `cohort` | category | `control` (pre-streaks, n=2,500) \| `streaks` (post-streaks, n=2,500) | | `retention_days_30` | int 0–30 | active days in first 30 days (zero-inflated, right-skewed) | | `plan` | category | `monthly \| annual` | | `acq_channel` | category | `organic \| paid \| referral` | ## 🎯 Required Deliverables 1. **`bootstrap_permutation.ipynb`** — EDA → bootstrap CI → permutation test → failure-mode demo. 2. **`retention_inference.csv`** — `statistic, estimate, ci_lo, ci_hi, method, n_boot` (median overall, median by cohort, difference-in-medians). 3. **`retention_memo.md`** — CI for the median, permutation p-value verdict on the lift, and one paragraph on when you would NOT bootstrap. 4. **`bootstrap_dist.png`** — bootstrap distribution of the median with CI bands marked. ## 🛠️ Step-by-Step Implementation Guide 1. **Show the t-interval failing.** Compute mean ± 1.96·SE on `retention_days_30`; overlay a normal curve on the skewed histogram to show the mismatch. Explain why the *median* is the right summary. 2. **Percentile bootstrap CI.** 10,000 resamples (seeded) of the median; 2.5/97.5 percentiles → 95% CI. Plot the bootstrap distribution with bands. 3. **Permutation test for the lift.** Shuffle `cohort` labels 10,000×, build the null distribution of difference-in-medians, exact p-value = fraction ≥ observed. Contrast with a naive t-test p-value. 4. **Break it on purpose.** Subsample n=15 tail quantiles (99th percentile) and show bootstrap instability; run an AR(1)-correlated series through naive bootstrap vs block bootstrap to demo dependence failure. 5. **Memo.** Numbers + verdict + limits (tails, dependence, small-n). ## 💻 Production Starter Scaffold ```python import numpy as np, pandas as pd rng = np.random.default_rng(11) def retention_mix(n, zero_p=0.35): zeros = rng.binomial(1, zero_p, n) vals = rng.negative_binomial(n=3, p=0.25, size=n).clip(0,30) return np.where(zeros, 0, vals) df = pd.DataFrame({ "user_id": [f"FT-{i:05d}" for i in range(5000)], "cohort": ["control"]*2500 + ["streaks"]*2500, "retention_days_30": np.concatenate([retention_mix(2500), retention_mix(2500)+rng.binomial(1,.12,2500)]), "plan": rng.choice(["monthly","annual"],5000,p=[.7,.3]), "acq_channel": rng.choice(["organic","paid","referral"],5000,p=[.5,.3,.2]), }) def bootstrap_ci(x, stat=np.median, n_boot=10_000, alpha=0.05, seed=11): r = np.random.default_rng(seed) boots = np.array([stat(r.choice(x, size=len(x), replace=True)) for _ in range(n_boot)]) return boots, tuple(np.percentile(boots, [100*alpha/2, 100*(1-alpha/2)])) def permutation_p(a, b, n_perm=10_000, seed=11): r = np.random.default_rng(seed) obs = np.median(b) - np.median(a) pool = np.concatenate([a,b]); na = len(a) null = np.array([np.median(r.permutation(pool)[na:]) - np.median(r.permutation(pool)[:na]) for _ in range(n_perm)]) return obs, float((np.abs(null) >= abs(obs)).mean()) med = df.retention_days_30.to_numpy() boots, ci = bootstrap_ci(med) print("median CI:", np.median(med), ci) a = df.query("cohort=='control'").retention_days_30.to_numpy() b = df.query("cohort=='streaks'").retention_days_30.to_numpy() print("lift, perm p:", permutation_p(a, b)) ``` ## 🧪 Acceptance Criteria & Verification Checks - [ ] t-interval-vs-histogram overlay visibly demonstrates the normality mismatch. - [ ] Median 95% CI from 10,000 seeded bootstraps; `bootstrap_dist.png` shows bands at 2.5/97.5%. - [ ] Permutation p-value computed from a 10,000-shuffle null (not a t-test) and reported with the observed lift. - [ ] Failure demo: tail-quantile instability + block-vs-naive bootstrap on correlated series, both shown. - [ ] Memo answers all three executive questions with numbers. --- ## 📊 Evaluation Rubric (100 points — passing ≥ 80) | # | Criterion | Points | What earns full marks | |---|---|---|---| | 1 | Statistical correctness | 25 | Right distributions/tests/CIs; assumptions checked, not assumed; math is reproducible | | 2 | Code quality & re