Case 5 — Diagnose Hallucinations and Add Guardrails

# Case 5 — Diagnose Hallucinations and Add Guardrails > *"Customer support is escalating complaints — the chatbot is inventing order details, promising refunds it can't deliver, and citing made-up policies. Diagnose and fix."* --- ## Step 1 — Confirm the problem Pull 10-20 recent complaints. Compare what the chatbot said to ground truth: ```sql SELECT request_id, customer_id, question, bot_answer, customer_complaint FROM support_tickets WHERE escalation_reason = 'incorrect_bot_response' AND created_at >= NOW() - INTERVAL '7 days' LIMIT 20; ``` Look for patterns. Common categories: 1. **Fabricated facts**: invented order numbers, tracking links, policies. 2. **Wrong commitments**: promising refunds outside policy. 3. **Misinterpretation**: wrong assumptions about customer intent. 4. **Outdated info**: model used training-data knowledge instead of current data. ## Step 2 — Find the root cause For each category, identify the underlying issue: ### Fabricated facts Likely cause: model relies on its training rather than retrieved context. Check: - Are retrieved chunks actually relevant? - Does the prompt clearly say "use ONLY the provided context"? - Is there an "I don't know" fallback? ### Wrong commitments Likely cause: tools that should gate actions don't. Check: - Is there a check_refund_eligibility tool? Is it being called? - Does the model have direct refund authority, or does it require human approval? ### Misinterpretation Likely cause: prompt doesn't push the model to clarify. Check: - Does the prompt encourage clarifying questions? - Is there an "ambiguous - escalate" path? ### Outdated info Likely cause: not using RAG, or RAG retrieval is poor. Check: - Is the knowledge base current? - Is retrieval surfacing relevant policy docs? ## Step 3 — Diagnose with a sample Take 5 problem cases. Manually run the same query through the pipeline. Inspect: ```python def debug_request(question, customer_id): print(f"Question: {question}") print() # Retrieval chunks = hybrid_search(question) print("RETRIEVED CHUNKS:") for i, c in enumerate(chunks): print(f" [{i}] {c['source_title']}: {c['text'][:200]}...") print() # Reranking reranked = rerank(question, chunks) print("RERANKED TOP 3:") for i, c in enumerate(reranked): print(f" [{i}] {c['source_title']}: {c['text'][:200]}...") print() # LLM response response = run_pipeline(question, customer_id) print(f"BOT ANSWER: {response['answer']}") print(f"CITATIONS: {response['sources']}") ``` For each problem case, you'll see one of: - Retrieval got the wrong chunks (fix retrieval). - Retrieval got the right chunks but the model ignored them (fix the prompt). - Right chunks + right prompt but model still hallucinated (fix the model or temperature). ## Step 4 — Apply guardrails ### Guardrail 1: Strengthen the "use only context" instruction ```python SYSTEM_PROMPT = '''You are a customer support assistant. CRITICAL: You must use ONLY the information in the provided context. - If the context doesn't contain the answer, you MUST say: "I don't have that information. Let me create a ticket for our support team." - NEVER invent order numbers, tracking links, policies, or commitments. - NEVER promise refunds, discounts, or returns without an explicit policy in the context. - ALWAYS cite the source for each claim you make. When you encounter: - Missing information → say "I don't know" and escalate. - Customer frustration → acknowledge and escalate. - Off-topic questions → politely redirect and escalate if needed. If unsure → escalate. Never guess.''' ``` Notice: explicit, repetitive instruction with "NEVER" language for high-stakes don'ts. ### Guardrail 2: Citation enforcement Require citations in every claim. If a response has uncited claims, flag for review: ```python def has_uncited_claims(answer): # Simple heuristic: every factual claim should have [Source X] sentences = answer.split(".") factual = [s for s in sentences if any(kw in s.lower() for kw in ["policy", "refund", "shipping", "tracking", "$"])] uncited = [s for s in factual if "[Source" not in s] return len(uncited) > 0 ``` If flagged: log, optionally re-prompt the model to add citations, or escalate to human review. ### Guardrail 3: Tool-gated actions Refunds, ticket creation, account changes — all gated through tools: ```python # Instead of allowing the model to "say" the customer will get a refund, # require it to call the refund tool. The tool checks eligibility. def initiate_refund(order_id, reason): eligibility = check_refund_eligibility(order_id) if not eligibility["eligible"]: return {"success": False, "reason": eligibility["reason"]} # ... initiate actual refund ``` The model can SAY "I've initiated a refund" only after the tool returns success=True. If the tool returns success=False, the model's response must reflect that. ### Guardrail 4: Output valida