SUBMISSION: Case 4 — Enterprise Hybrid RAG with Cross-Encoder Reranking

# SUBMISSION: Case 4 — Enterprise RAG Architecture: Build a Production Retrieval Pipeline with Hybrid Search & Cross-Encoder Reranking > **Industry Context:** A technical documentation assistant for a Kubernetes cloud platform must answer complex engineer inquiries across 500 internal markdown guides. Pure dense semantic vector search fails on exact CLI flags (`--kubeconfig`, `--dry-run=client`) and error codes, while pure BM25 keyword search misses conceptual queries. Build an enterprise hybrid search pipeline. --- ## Deliverables 1. `ingest_pipeline.py`: Document loader and recursive chunker (chunk size 512, overlap 64) embedding documents into Chroma/pgvector while building an inverted BM25 index. 2. `hybrid_retriever.py`: Hybrid search engine combining dense vector cosine similarity and BM25 scores via Reciprocal Rank Fusion (RRF, k=60), followed by a cross-encoder reranker to yield the top-5 passages. 3. `rag_generator.py`: Prompt engine generating answers with grounded source citations (`[Doc X, Section Y]`) and an explicit "I don't have enough information" fallback when relevance falls below threshold. 4. `rag_eval.py`: Evaluation script measuring context retrieval recall and answer faithfulness. --- ## Starter Code Scaffold ```python from typing import List, Dict, Any def reciprocal_rank_fusion(dense_results: List[str], bm25_results: List[str], k: int = 60) -> List[tuple]: """Combine dense and sparse ranked lists using Reciprocal Rank Fusion (RRF).""" rrf_score = {} for rank, doc_id in enumerate(dense_results): rrf_score[doc_id] = rrf_score.get(doc_id, 0.0) + 1.0 / (k + rank + 1) for rank, doc_id in enumerate(bm25_results): rrf_score[doc_id] = rrf_score.get(doc_id, 0.0) + 1.0 / (k + rank + 1) return sorted(rrf_score.items(), key=lambda x: x[1], reverse=True) ``` --- ## 100-Point Evaluation Rubric - **Criterion 1: Ingestion & Chunking Strategy (25 pts)** — Document chunking with clean sentence boundary preservation, metadata indexing, dimension consistency. - **Criterion 2: Hybrid Retrieval & RRF Fusion (25 pts)** — Dual retrieval pipeline marrying dense vector embeddings and BM25 sparse index via Reciprocal Rank Fusion. - **Criterion 3: Cross-Encoder Reranking (25 pts)** — Second-stage scoring with cross-encoder reranker to compress top 20 candidates into top 5 high-density chunks. - **Criterion 4: Faithfulness, Citations & Fallback (25 pts)** — Traceable source citations (`[Doc, Section]`) and strict "I don't know" fallback on out-of-domain queries. --- ## Implementation Guide & Case Steps > *"Build a chatbot that answers questions from a corpus of 500 support documents. Production-ready: hybrid search, reranking, citations, 'I don't know' fallback, evaluation."* --- ## Step 1 — Architecture overview ``` User question ↓ [Embed query] ↓ [Hybrid search (dense + BM25)] → top 20 candidates ↓ [Cross-encoder rerank] → top 5 ↓ [LLM with retrieved context + citations + "I don't know" fallback] ↓ Answer with citations ``` 5 components. Let's build each. ## Step 2 — Document ingestion ```python from langchain.text_splitter import RecursiveCharacterTextSplitter def ingest_documents(docs): splitter = RecursiveCharacterTextSplitter( chunk_size=500, chunk_overlap=50, separators=["\n\n", "\n", ". ", " ", ""], ) chunks = [] for doc in docs: for chunk_text in splitter.split_text(doc.text): chunks.append({ "text": chunk_text, "metadata": { "source_id": doc.id, "source_title": doc.title, "source_url": doc.url, "ingested_at": datetime.utcnow().isoformat(), } }) return chunks # Embed and index from openai import OpenAI client = OpenAI() def embed_chunks(chunks): texts = [c["text"] for c in chunks] response = client.embeddings.create( model="text-embedding-3-large", input=texts, ) for chunk, embedding_data in zip(chunks, response.data): chunk["embedding"] = embedding_data.embedding return chunks # Store in Qdrant from qdrant_client import QdrantClient from qdrant_client.models import VectorParams, Distance, PointStruct qdrant = QdrantClient(host="localhost", port=6333) qdrant.recreate_collection( collection_name="support_docs", vectors_config=VectorParams(size=3072, distance=Distance.COSINE), ) def index_chunks(chunks): points = [ PointStruct( id=i, vector=chunk["embedding"], payload={"text": chunk["text"], **chunk["metadata"]} ) for i, chunk in enumerate(chunks) ] qdrant.upsert(collection_name="support_docs", points=points) ``` ## Step 3 — Hybrid search ```python from rank_bm25 import BM25Okapi import re # Build BM25 index tokenized_corpus = [re.findall(r"\w+", c["text"].lower()) for c in chunks] bm25 = BM25Okapi(t