Status: UNPUBLISHED draft. Do not enroll learners until the P4 milestone placeholder (data-scientist track, RAG bot milestone, is_required=False) is flipped to required and the eval-set below is seeded as an unpublished practice test. Ref: DS track seed backend/scripts/seed_career_track_data_scientist.py.

1. Objective

Build a working retrieval-augmented generation (RAG) chatbot over a messy, realistic business document corpus (support tickets + policy PDFs + product wiki pages with duplicates, stale versions, and contradictory passages). The bot must answer grounded in retrieved passages, cite sources, refuse cleanly when the corpus has no answer, and stay within a per-query cost/latency budget.

2. Deliverables

Ingestion pipeline: chunking + embeddings + vector index build script (re-runnable; committed eval/ config pins chunk size, overlap, model).
Query service: POST /ask {question} → {answer, citations[], refused}.
Eval report: 20Q golden suite results (section 5) with groundedness, refusal, and cost/latency numbers.
Short write-up (≤2 pages): chunking choices, failure cases, what you would change with 10× corpus size.

3. Constraints

Every non-refused answer must carry ≥1 citation to a retrieved chunk id.
Out-of-corpus questions MUST be refused (refused=true), never hallucinated.
p95 end-to-end latency ≤ 8 s; mean cost per query ≤ $0.02 (log both).
No fine-tuning required; prompt + retrieval engineering only.

4. 100-Point Rubric

#	Criterion	Points
R1	Ingestion is re-runnable; chunk/overlap/embedding choices documented and justified	10
R2	Retrieval quality: recall@5 ≥ 0.80 on the 20Q golden suite	20
R3	Groundedness: ≥ 90% of non-refused answers fully supported by cited chunks	20
R4	Refusal discipline: 100% refusal on the 5 out-of-corpus probes, 0 false refusals on answerable Qs	15
R5	Citations: correct chunk ids on ≥ 90% of non-refused answers	10
R6	Cost/latency: p95 ≤ 8 s and mean ≤ $0.02/query with logged evidence	10
R7	Failure analysis: ≥ 3 real failure cases reproduced with fix proposals	10
R8	Write-up clarity + reproducibility (cold-clone run by reviewer)	5
	Total	100

Pass: ≥ 80 overall AND R4 = 15/15 (refusal is a hard gate).

5. Eval-Set Spec — 20Q Golden Suite

Stored as unpublished practice test slug ds-rag-golden-20q (is_published=False, type mixed). Composition:

8 factual lookups (answer in exactly one passage; tests chunking/recall).
4 multi-hop questions (answer needs ≥ 2 passages; tests top-k + synthesis).
3 stale/contradiction questions (two passages disagree; correct answer cites the newer version or flags the conflict).
5 out-of-corpus probes (correct behavior = refusal; any answer = fail).

Each item records: question, answerable (bool), gold_chunk_ids[], gold_answer (for answerable), refuse_expected (for probes). Metrics: recall@5, groundedness rate (judge: supported / unsupported / refused), refusal precision/recall, p95 latency, mean cost/query.

6. Cost-Latency Logging Spec

Every /ask call appends {q, ms, input_tokens, output_tokens, est_cost_usd, top_k, refused} to eval/run_log.jsonl. The eval report aggregates p50/p95 latency and mean cost. Budget: p95 ≤ 8000 ms, mean ≤ $0.02.

Eval corpus (authored, versioned with this brief)

RAG-C01 [wiki] — Resetting your password

To reset your password in AcmeDesk: open Settings > Security > Reset password. You receive a one-time link valid for 30 minutes. Links expire after one use.

RAG-C02 [ticket] — Ticket #4812 — CSV export fails on large workspaces

Customer export of 62,000 rows failed with a timeout. Support confirmed CSV export is limited to 50,000 rows per file. Workaround: split the export by date range so each file stays under 50,000 rows, then concatenate.

RAG-C03 [policy] — Refund Policy v2 (current, effective 2026-08-01)

Current refund policy: full refund within 14 days of purchase. Annual plans cancelled after 14 days are refunded pro-rata for unused full months. To request a refund, contact billing@acmedesk.example from the workspace owner email.

RAG-C04 [policy] — Refund Policy v1 (STALE, superseded 2026-08-01)

Old refund policy (no longer in force): full refund within 30 days of purchase. Annual plans were non-refundable after purchase. Kept for audit history only.

RAG-C05 [wiki] — Setting up SSO with Okta

SSO setup: in AcmeDesk go to Settings > SSO, copy the ACS URL and Entity ID into your Okta app, then paste the Okta metadata URL back into AcmeDesk and click Verify. Assign users in Okta before they first sign in.

RAG-C06 [wiki] — Which plans include SSO

SSO (SAML/OIDC) is available on Growth plans and above. Starter and Free workspaces cannot enable SSO; they use password plus optional two-factor authentication.

RAG-C07 [pricing] — Pricing page v3 (current, published 2026-09-01)

Current per-seat pricing: Starter $19, Growth $49, Enterprise custom. Annual billing saves 20% versus monthly. Prices in USD, exclusive of tax.

RAG-C08 [pricing] — Pricing page v2 (STALE, replaced 2026-09-01)

Old per-seat pricing: Starter $15, Growth $39, Enterprise custom. Do not quote; retained for changelog reference.

RAG-C09 [ticket] — Ticket #5120 — API rate limits

The public API allows 600 requests per minute per token with burst up to 50 extra requests. Exceeding the limit returns HTTP 429 with a Retry-After header. There is no per-day quota on paid plans.

RAG-C10 [wiki] — Data retention and workspace deletion

Deleted workspaces are retained for 90 days, then permanently purged. Export any data you need before deleting: Settings > Danger zone > Export, then Delete workspace.

RAG-C11 [wiki] — Slack integration and channel alerts

Connect Slack via Settings > Integrations > Slack. Channel alerts support ticket created, ticket resolved, and mention events. Alert delivery is typically under 1 minute.

RAG-C12 [status] — Status page — uptime SLO

AcmeDesk targets 99.9% monthly uptime for the app and API, excluding scheduled maintenance windows announced at least 72 hours in advance on the status page.