SUBMISSION: Capstone 2 — Bayesian A/B Testing Engine with Conjugate Beta-Binomial Updating

🎯 Capstone Portfolio Deliverable. Ship a decision engine, not a one-off analysis — the artifact a growth team runs every week.

🏢 Business Scenario & Executive Prompt

LearnLoop (ed-tech) split its onboarding flow: old flow (years of history, 7.0% activation) vs new flow (2 weeks live, 800 users, 8.0% activation). Frequentist test: p = 0.31 — "not significant, keep waiting." Growth wants a decision now: the new flow costs eng upkeep every week it stays in test mode. The CPO asks: "What is the probability the new flow is better, how much do we lose if we are wrong, and at what threshold should we ship?"

📋 Source Data & Input Schemas

learnloop_onboarding.csv — daily cumulative snapshots, 14 days (mock shape; scaffold generates it):

Prior context: old flow 7.0% over ~400K users (strong prior available); new flow unknown (weak/flat prior). Stakes: each extra test week costs ~₹3.5L in eng + delayed growth; shipping a worse flow costs ~1% activation ≈ ₹28L/quarter.

🎯 Required Deliverables

1. bayesian_engine.py — update_beta(), prob_b_beats_a(n_samples), expected_loss(), ship_recommendation() (importable module, not just a notebook).
2. bayesian_ab.ipynb — engine demo: day-by-day posterior animation data, prior-sensitivity grid, threshold analysis.
3. posterior_day14.png — overlapping Beta posteriors + shaded P(new > old) region.
4. bayes_memo.md — P(new>old), expected loss of shipping vs waiting, ship/wait verdict with threshold rule.

🛠️ Step-by-Step Implementation Guide

1. Conjugate updating. Old: Beta(0.07·400K, 0.93·400K)-equivalent informative prior updated with test data. New: flat Beta(1,1) updated with 800 users. Plot both posteriors.
2. P(new > old | data). Monte-Carlo: 200K draws from each posterior, fraction where new > old. Track it day-by-day (1–14) to show the decision trajectory frequentist peeking cannot give.
3. Expected-loss ship rule. Loss(ship) = E[max(0, θ_old − θ_new)]; Loss(wait) = weekly cost. Ship when Loss(ship) < Loss(wait). Report both in rupees.
4. Prior sensitivity. Grid over new-flow priors {flat, skeptical Beta(7,93), enthusiastic Beta(14,86)}: does the verdict flip? State the robustness conclusion.
5. Engine + memo. Package steps 1–3 as functions; memo gives the threshold rule the team reuses ("ship when P>90% AND expected loss < ₹1L").

💻 Production Starter Scaffold

import numpy as np

def update_beta(a0, b0, conv, n):
    return a0 + conv, b0 + (n - conv)

def prob_b_beats_a(aA, bA, aB, bB, n_samples=200_000, seed=0):
    r = np.random.default_rng(seed)
    return float((r.beta(aB, bB, n_samples) > r.beta(aA, bA, n_samples)).mean())

def expected_loss_ship(aA, bA, aB, bB, n_samples=200_000, seed=1):
    r = np.random.default_rng(seed)
    return float(np.maximum(0, r.beta(aA, bA, n_samples) - r.beta(aB, bB, n_samples)).mean())

# Old flow: strong prior ~7% ; New flow: flat prior; day-14 data
aA, bA = update_beta(0.07*400_000, 0.93*400_000, 2240, 32_000)
aB, bB = update_beta(1, 1, 64, 800)
print("P(new>old):", round(prob_b_beats_a(aA,bA,aB,bB), 3))
print("exp loss/ship (rate pts):", round(expected_loss_ship(aA,bA,aB,bB), 4))

🧪 Acceptance Criteria & Verification Checks

• bayesian_engine.py imports cleanly with the 4 required functions (autograder imports it).
• Day-by-day P(new>old) trajectory computed (14 values) and plotted or tabled.
• Expected-loss ship rule evaluated in rupees against the weekly waiting cost.
• Prior-sensitivity grid (≥3 priors) with a stated robustness verdict.
• posterior_day14.png shows both posteriors + shaded superiority region.

📊 Evaluation Rubric (100 points — passing ≥ 80)

📤 Submission Guidelines

1. Upload your files (.ipynb, .csv, .md, .png) via the file dropzone on this lesson.
2. Paste your public GitHub repo URL in the submission notes (repo must contain the same files + a 5-line README with the run command).
3. Name files exactly as listed under Required Deliverables — autograding checks filenames first.
4. Passing score is 80/100. You may resubmit; the highest score counts.