Recap — Building a Working RAG Pipeline

# Building a Working RAG Pipeline RAG = Retrieval-Augmented Generation. At query time, retrieve relevant context from your data, give it to the LLM, and let the LLM answer using that context. It's the most-used pattern in production LLM apps because it solves the biggest LLM limitation: the model doesn't know your specific data. ## The 5-piece pipeline ``` 1. Documents → Chunking → Chunks 2. Chunks → Embedding model → Vectors 3. Vectors → Vector database 4. User query → Embedding → Similarity search → Retrieved chunks 5. Query + retrieved chunks → LLM → Answer ``` Each piece has design decisions. Let's walk through them. ## Step 1 — Chunking Documents (PDFs, web pages, support tickets, etc.) get split into chunks. Why chunk at all? - Embeddings represent fixed-size text. - LLMs have context limits. - Smaller chunks = more precise retrieval. Chunking strategies: ### Fixed-size chunks Split every N characters or tokens. Simple. Often the right starting point. ```python chunk_size = 500 # tokens or characters overlap = 50 # to prevent splitting concepts at chunk boundaries ``` ### Sentence/paragraph-aware chunks Split at natural boundaries (sentences, paragraphs) up to a max size. Better preserves semantic units. ### Hierarchical chunks Multiple sizes (sentence, paragraph, section). Retrieve at appropriate granularity. ### Semantic chunking Use embeddings to split where topic shifts. Sophisticated; mostly research-grade. Practical default: paragraph-aware chunking with 400-800 token chunks and 50-token overlap. Tune based on your data. ## Step 2 — Embeddings Each chunk gets converted to a vector via an embedding model. Popular embedding models in 2026: - `text-embedding-3-large` (OpenAI) — strong general-purpose. - `voyage-3` (Voyage AI) — specialized for RAG, often outperforms. - `BGE-M3` (BAAI) — open-source, strong multilingual. - `jina-embeddings-v3` (Jina) — open-source, long context. Considerations: - **Dimensions**: typically 768-3072. Smaller = faster, less storage. Larger = slightly better. - **Cost**: API embedding models cost a fraction of LLM calls but add up over millions of chunks. - **Domain**: general models work for most. Specialized models (legal, medical) can boost performance. ## Step 3 — Vector database Store embeddings somewhere queryable. Options: ### Hosted services - **Pinecone** — most popular, managed. - **Qdrant Cloud** — open-source core, managed offering. - **Weaviate** — open-source, managed offering. - **Voyage AI / Cohere / Vercel** — bundled RAG services. ### Self-hosted - **pgvector** — PostgreSQL extension. Works for moderate scale; you already have Postgres. - **Qdrant** — self-hostable. - **ChromaDB** — local-first, easy to start with. - **Milvus** — for very large scale. For most projects under 10M vectors, pgvector or Qdrant is enough. Pinecone if you want zero ops. ## Step 4 — Retrieval Given a user query: 1. Embed the query with the same model used for chunks. 2. Search the vector DB for nearest neighbors (cosine similarity is standard). 3. Return top-K most similar chunks. ```python query_embedding = embed(user_query) results = vector_db.search(query_embedding, top_k=5) retrieved_chunks = [r.text for r in results] ``` Tuning: - **top_k**: typically 3-10. More chunks = more context but more "lost in the middle" risk. - **score threshold**: minimum similarity; filter out weak matches. ## Step 5 — Generation Construct a prompt that includes retrieved chunks: ```python prompt = f''' Answer the user's question based on the following context. If the context doesn't contain the answer, say "I don't know". Context: {chr(10).join(retrieved_chunks)} Question: {user_query} Answer: ''' ``` The model uses the retrieved context to answer. "I don't know" fallback prevents hallucination when retrieval misses. ## What goes wrong (the common failures) ### Failure 1: Poor retrieval The vector DB returns irrelevant chunks. Common causes: - Bad chunking (concepts split across chunks). - Embedding model mismatch with content (general model for specialized domain). - Query/document mismatch (user asks in natural language; documents are in technical jargon). Fix: try different embedding models; use hybrid search (next section); rerank. ### Failure 2: The LLM ignores retrieved context The model answers from its general knowledge instead of the provided context. Causes: - Context is buried in a long prompt (lost in the middle). - Instructions don't emphasize using ONLY the provided context. Fix: structure the prompt clearly; explicit instruction to use only the context; lower temperature. ### Failure 3: Hallucination despite RAG Model makes up facts not in the retrieved chunks. Causes: - Retrieval brought in tangentially related content. - Model fills gaps when retrieval is incomplete. Fix: prompt instructs to say "I don't know" when unsure; raise the score threshold; add citations. ## Hybrid search (the upgrade) Pure vector search struggles w