SUBMISSION: Capstone 2 — Bayesian A/B Testing Engine with Conjugate Beta-Binomial Updating

# SUBMISSION: Capstone 2 — Bayesian A/B Testing Engine with Conjugate Beta-Binomial Updating > 🎯 **Capstone Portfolio Deliverable.** Ship a *decision engine*, not a one-off analysis — the artifact a growth team runs every week. ## 🏢 Business Scenario & Executive Prompt **LearnLoop** (ed-tech) split its onboarding flow: old flow (years of history, 7.0% activation) vs new flow (2 weeks live, 800 users, 8.0% activation). Frequentist test: p = 0.31 — *"not significant, keep waiting."* Growth wants a decision *now*: the new flow costs eng upkeep every week it stays in test mode. The CPO asks: *"What is the probability the new flow is better, how much do we lose if we are wrong, and at what threshold should we ship?"* ## 📋 Source Data & Input Schemas **`learnloop_onboarding.csv`** — daily cumulative snapshots, 14 days (mock shape; scaffold generates it): | column | dtype | description | |---|---|---| | `day` | int 1–14 | days since test start | | `n_old`, `conv_old` | int | cumulative users / activations, old flow | | `n_new`, `conv_new` | int | cumulative users / activations, new flow | Prior context: old flow 7.0% over ~400K users (strong prior available); new flow unknown (weak/flat prior). Stakes: each extra test week costs ~₹3.5L in eng + delayed growth; shipping a worse flow costs ~1% activation ≈ ₹28L/quarter. ## 🎯 Required Deliverables 1. **`bayesian_engine.py`** — `update_beta()`, `prob_b_beats_a(n_samples)`, `expected_loss()`, `ship_recommendation()` (importable module, not just a notebook). 2. **`bayesian_ab.ipynb`** — engine demo: day-by-day posterior animation data, prior-sensitivity grid, threshold analysis. 3. **`posterior_day14.png`** — overlapping Beta posteriors + shaded P(new > old) region. 4. **`bayes_memo.md`** — P(new>old), expected loss of shipping vs waiting, ship/wait verdict with threshold rule. ## 🛠️ Step-by-Step Implementation Guide 1. **Conjugate updating.** Old: Beta(0.07·400K, 0.93·400K)-equivalent informative prior updated with test data. New: flat Beta(1,1) updated with 800 users. Plot both posteriors. 2. **P(new > old | data).** Monte-Carlo: 200K draws from each posterior, fraction where new > old. Track it day-by-day (1–14) to show the *decision trajectory* frequentist peeking cannot give. 3. **Expected-loss ship rule.** Loss(ship) = E[max(0, θ_old − θ_new)]; Loss(wait) = weekly cost. Ship when Loss(ship) 90% AND expected loss r.beta(aA, bA, n_samples)).mean()) def expected_loss_ship(aA, bA, aB, bB, n_samples=200_000, seed=1): r = np.random.default_rng(seed) return float(np.maximum(0, r.beta(aA, bA, n_samples) - r.beta(aB, bB, n_samples)).mean()) # Old flow: strong prior ~7% ; New flow: flat prior; day-14 data aA, bA = update_beta(0.07*400_000, 0.93*400_000, 2240, 32_000) aB, bB = update_beta(1, 1, 64, 800) print("P(new>old):", round(prob_b_beats_a(aA,bA,aB,bB), 3)) print("exp loss/ship (rate pts):", round(expected_loss_ship(aA,bA,aB,bB), 4)) ``` ## 🧪 Acceptance Criteria & Verification Checks - [ ] `bayesian_engine.py` imports cleanly with the 4 required functions (autograder imports it). - [ ] Day-by-day P(new>old) trajectory computed (14 values) and plotted or tabled. - [ ] Expected-loss ship rule evaluated in rupees against the weekly waiting cost. - [ ] Prior-sensitivity grid (≥3 priors) with a stated robustness verdict. - [ ] `posterior_day14.png` shows both posteriors + shaded superiority region. --- ## 📊 Evaluation Rubric (100 points — passing ≥ 80) | # | Criterion | Points | What earns full marks | |---|---|---|---| | 1 | Statistical correctness | 25 | Right distributions/tests/CIs; assumptions checked, not assumed; math is reproducible | | 2 | Code quality & reproducibility | 25 | Seeded RNG, clean re-runnable notebook, typed schemas, no hardcoded magic numbers | | 3 | Business reasoning & memo | 25 | Decision-oriented memo with costed trade-offs; honest about uncertainty and limits | | 4 | Visualization & communication | 25 | Labeled, honest axes (log-scale where skewed); figures referenced in the memo | ## 📤 Submission Guidelines 1. Upload your files (`.ipynb`, `.csv`, `.md`, `.png`) via the file dropzone on this lesson. 2. Paste your public GitHub repo URL in the submission notes (repo must contain the same files + a 5-line `README` with the run command). 3. Name files exactly as listed under Required Deliverables — autograding check