Capstone Brief: RAG Chatbot on a Messy Business Corpus (UNPUBLISHED)

# DS Project Brief 1 — RAG Chatbot on a Messy Business Corpus (UNPUBLISHED DRAFT) > Status: UNPUBLISHED draft. Do not enroll learners until the P4 milestone > placeholder (`data-scientist` track, RAG bot milestone, `is_required=False`) > is flipped to required and the eval-set below is seeded as an unpublished > practice test. Ref: DS track seed `backend/scripts/seed_career_track_data_scientist.py`. ## 1. Objective Build a working retrieval-augmented generation (RAG) chatbot over a messy, realistic business document corpus (support tickets + policy PDFs + product wiki pages with duplicates, stale versions, and contradictory passages). The bot must answer grounded in retrieved passages, cite sources, refuse cleanly when the corpus has no answer, and stay within a per-query cost/latency budget. ## 2. Deliverables 1. Ingestion pipeline: chunking + embeddings + vector index build script (re-runnable; committed `eval/` config pins chunk size, overlap, model). 2. Query service: `POST /ask {question}` → `{answer, citations[], refused}`. 3. Eval report: 20Q golden suite results (section 5) with groundedness, refusal, and cost/latency numbers. 4. Short write-up (≤2 pages): chunking choices, failure cases, what you would change with 10× corpus size. ## 3. Constraints - Every non-refused answer must carry ≥1 citation to a retrieved chunk id. - Out-of-corpus questions MUST be refused (`refused=true`), never hallucinated. - p95 end-to-end latency ≤ 8 s; mean cost per query ≤ $0.02 (log both). - No fine-tuning required; prompt + retrieval engineering only. ## 4. 100-Point Rubric | # | Criterion | Points | |---|-----------|--------| | R1 | Ingestion is re-runnable; chunk/overlap/embedding choices documented and justified | 10 | | R2 | Retrieval quality: recall@5 ≥ 0.80 on the 20Q golden suite | 20 | | R3 | Groundedness: ≥ 90% of non-refused answers fully supported by cited chunks | 20 | | R4 | Refusal discipline: 100% refusal on the 5 out-of-corpus probes, 0 false refusals on answerable Qs | 15 | | R5 | Citations: correct chunk ids on ≥ 90% of non-refused answers | 10 | | R6 | Cost/latency: p95 ≤ 8 s and mean ≤ $0.02/query with logged evidence | 10 | | R7 | Failure analysis: ≥ 3 real failure cases reproduced with fix proposals | 10 | | R8 | Write-up clarity + reproducibility (cold-clone run by reviewer) | 5 | | | **Total** | **100** | Pass: ≥ 80 overall AND R4 = 15/15 (refusal is a hard gate). ## 5. Eval-Set Spec — 20Q Golden Suite Stored as unpublished practice test slug `ds-rag-golden-20q` (`is_published=False`, type `mixed`). Composition: - 8 factual lookups (answer in exactly one passage; tests chunking/recall). - 4 multi-hop questions (answer needs ≥ 2 passages; tests top-k + synthesis). - 3 stale/contradiction questions (two passages disagree; correct answer cites the newer version or flags the conflict). - 5 out-of-corpus probes (correct behavior = refusal; any answer = fail). Each item records: `question`, `answerable` (bool), `gold_chunk_ids[]`, `gold_answer` (for answerable), `refuse_expected` (for probes). Metrics: recall@5, groundedness rate (judge: supported / unsupported / refused), refusal precision/recall, p95 latency, mean cost/query. ## 6. Cost-Latency Logging Spec Every `/ask` call appends `{q, ms, input_tokens, output_tokens, est_cost_usd, top_k, refused}` to `eval/run_log.jsonl`. The eval report aggregates p50/p95 latency and mean cost. Budget: p95 ≤ 8000 ms, mean ≤ $0.02. --- # Eval corpus (authored, versioned with this brief) ### RAG-C01 [wiki] — Resetting your password To reset your password in AcmeDesk: open Settings > Security > Reset password. You receive a one-time link valid for 30 minutes. Links expire after one use. ### RAG-C02 [ticket] — Ticket #4812 — CSV export fails on large workspaces Customer export of 62,000 rows failed with a timeout. Support confirmed CSV export is limited to 50,000 rows per file. Workaround: split the export by date range so each file stays under 50,000 rows, then concatenate. ### RAG-C03 [policy] — Refund Policy v2 (current, effective 2026-08-01) Current refund policy: full refund within 14 days of purchase. Annual plans cancelled after 14 days are refunded pro-rata for unused full months. To request a refund, contact billing@acmedesk.example from the workspace owner email. ### RAG-C04 [policy] — Refund Policy v1 (STALE, superseded 2026-08-01) Old refund policy (no longer in force): full refund within 30 days of purchase. Annual plans were non-refundable after purchase. Kept for audit history only. ### RAG-C05 [wiki] — Setting up SSO with Okta SSO setup: in AcmeDesk go to Settings > SSO, copy the ACS URL and Entity ID into your Okta app, then paste the Okta metadata URL back into AcmeDesk and click Verify. Assign users in Okta before they first sign in. ### RAG-C06 [wiki] — Which plans include SSO SSO (SAML/OIDC) is available on Growth plans and above. Starter and Free workspaces cannot enable SSO; they use password plus optiona