RAG Architecture + Trade-offs

Outcome: Choose RAG vs fine-tune for 3 cases Curated video (IBM Technology): What is Retrieval-Augmented Generation (RAG)? — https://www.youtube.com/watch?v=T-D1OfcDW1M (verified live via yt-dlp 2026-09-24). Pointer: llms-genai-for-practitioners/12 (hybrid legal-AI pattern); shell: courses/video-scripts/genai-rag-agents/03.md.

7 minutesVideo LessonPDF notes

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Ready to continue?

Mark this lesson as complete when you're ready to proceed.

Key moments

Introduction to RAG — Retrieval Augmented Generation (RAG) helps Large Language Models be more accurate and current.
LLM Challenges Defined — Standalone LLMs suffer from lack of source and out-of-date information.
Grounding Example — Looking up facts on a reputable source prevents hallucination and ensures accuracy.
LLM vs RAG Response — A standalone LLM confidently answers based on potentially outdated training data.
RAG Architecture — RAG adds a retriever and a content store to the LLM process, moving beyond internal knowledge.
RAG Process Flow — The RAG framework instructs the LLM to retrieve relevant content before generating a final answer.
RAG Solves Timeliness — RAG addresses out-of-date information by updating the external data store instead of retraining the model.
RAG Solves Sourcing — RAG forces the model to use primary source data, reducing hallucination and enabling evidence provision.
RAG Trade-offs — If the retriever fails to find quality data, the model may incorrectly respond with 'I don't know' even if the answer exists.

PDF notes

Bottom line: You will be able to identify the core limitations of standalone Large Language Models (LLMs) regarding sourcing and timeliness. You will learn how Retrieval Augmented Generation (RAG) architecture addresses these issues by integrating external knowledge stores. This allows you to explain the trade-offs involved in designing effective RAG systems.

This lesson on RAG Architecture + Trade-offs is hands-on and example-driven. You will be able to identify the core limitations of standalone Large Language Models (LLMs) regarding sourcing and timeliness. You will learn how Retrieval Augmented Generation (RAG) architecture addresses these issues by integrating external knowledge stores. This allows you to explain the trade-offs involved in designing effective RAG systems.

What You'll Be Able To Do

Identify the two primary challenges inherent in standalone Large Language Models (LLMs).
Explain the role of the retriever and the content store in the RAG framework.
Contrast how RAG handles new information versus traditional LLM retraining.
Describe the three components required in a RAG-enhanced prompt.
Evaluate the critical dependency of RAG performance on retriever quality.
Determine scenarios where a RAG model should respond with "I don't know."

Topics Covered in RAG Architecture + Trade-offs

Introduction to RAG (0:00 - 0:35) — Retrieval Augmented Generation (RAG) helps Large Language Models be more accurate and current.
LLM Challenges Defined (0:35 - 1:50) — Standalone LLMs suffer from lack of source and out-of-date information.
Grounding Example (1:50 - 2:45) — Looking up facts on a reputable source prevents hallucination and ensures accuracy.
LLM vs RAG Response (2:45 - 3:30) — A standalone LLM confidently answers based on potentially outdated training data.
RAG Architecture (3:30 - 4:15) — RAG adds a retriever and a content store to the LLM process, moving beyond internal knowledge.
RAG Process Flow (4:15 - 5:00) — The RAG framework instructs the LLM to retrieve relevant content before generating a final answer.
RAG Solves Timeliness (5:00 - 5:50) — RAG addresses out-of-date information by updating the external data store instead of retraining the model.
RAG Solves Sourcing (5:50 - 6:40) — RAG forces the model to use primary source data, reducing hallucination and enabling evidence provision.
RAG Trade-offs (6:40 - 7:30) — If the retriever fails to find quality data, the model may incorrectly respond with 'I don't know' even if the answer exists.

SQL Cheat Sheet

Retrieval Augmented Generation (RAG) — Framework to ground LLM answers using external, verifiable data
```
SELECT * FROM knowledge_base WHERE topic = 'moons';
```
LLM Challenge: No Source — Model generates text without verifiable evidence (hallucination)
```
SELECT answer FROM llm_parameters WHERE confidence > 0.9;
```
LLM Challenge: Out of Date — Answers rely on static training data, missing recent facts
```
UPDATE content_store SET data = 'Saturn 146 moons' WHERE fact_id = 101;
```
Content Store — External data source providing grounding information for the LLM
```
SELECT document_text FROM policies WHERE department = 'HR';
```

Retriever — Component that finds relevant content based on the user's query

SELECT relevant_chunks FROM vector_index ORDER BY similarity DESC LIMIT 5;

Grounded Answer — Response supported by evidence from the external content store

SELECT response, evidence_source FROM rag_output WHERE is_grounded = TRUE;

Comparison Table

Feature	Standalone LLM	RAG Framework
Knowledge Source	Internal training parameters	External content store
Handling New Data	Requires full model retraining	Update external data store
Risk of Hallucination	High (no source)	Low (grounded in retrieval)
Answer Confidence	Confident, even if wrong	Grounded, can say "I don't know"

Common Pitfalls

Mistake: Assuming RAG eliminates all hallucination risk. Avoid: Ensure the retriever provides high-quality, relevant grounding data.
Mistake: Relying on the LLM's internal knowledge when the content store is incomplete. Avoid: Instruct the model to prioritize retrieved content over trained parameters.
Mistake: Thinking RAG only solves the 'out-of-date' problem. Avoid: Recognize RAG primarily solves both sourcing and timeliness issues simultaneously.
Mistake: Ignoring the quality of the retrieval step. Avoid: Invest effort in improving the retriever component alongside the generator.

FAQs

Why is RAG better than just retraining the LLM frequently? Retraining is expensive and time-consuming. RAG allows instant updates simply by modifying the external content store.
What does it mean for an answer to be "grounded"? A grounded answer is supported by specific, verifiable evidence retrieved from the external content store. This prevents the model from making up facts.
What are the three parts of a RAG-enhanced prompt? The prompt includes the user's question, the instruction to pay attention to retrieved content, and the retrieved content itself.
Can RAG leak personal or sensitive information? RAG reduces the likelihood of leaking data learned during training, but the content store itself must be secured and curated.

Common pitfalls

Mistake: Assuming RAG eliminates all hallucination risk. Avoid: Ensure the retriever provides high-quality, relevant grounding data.
Mistake: Relying on the LLM's internal knowledge when the content store is incomplete. Avoid: Instruct the model to prioritize retrieved content over trained parameters.
Mistake: Thinking RAG only solves the 'out-of-date' problem. Avoid: Recognize RAG primarily solves both sourcing and timeliness issues simultaneously.
Mistake: Ignoring the quality of the retrieval step. Avoid: Invest effort in improving the retriever component alongside the generator.

Large language models, they are everywhere. They get some things amazingly right, and other things very interestingly wrong. My name is Marina Danilevsky. I am a Senior Research Scientist here at IBM Research, and I want to tell you about a framework to help large language models be more accurate and more up-to-date: Retrieval Augmented Generation, or RAG. Let's just talk about the generation part for a minute. So, forget the retrieval augmented. So, the generation, this refers to large language models, or LLMs, that generate text in response to a user query, referred to as a prompt. These models can have some undesirable behavior. I want to tell you an anecdote uh to illustrate this. So, my kids, they recently asked me this question: "In our solar system, what planet has the most moons?" And my response was, "Oh, that's really great that you're asking me this question. I loved space when I was your age. Of course, that was like 30 years ago, but I know this. I read an article, and the article said that it was Jupiter and 88 moons, so that's the answer." Now, actually, there's a couple of things wrong with my answer. First of all, uh I have no source to support what I'm saying. So, even though I confidently said, "I read an article, I know the answer," I'm not sourcing it. I'm giving the answer off the top of my head. And also, I actually haven't kept up with this for a while, and my answer is out of date. So, we have two problems here. One is no source. And the second problem is that I am out of date. And these, in fact, are two behaviors that are often observed as problematic when interacting with large language models. They are LLM challenges. Now, what would have happened if I'd taken a beat and first gone and looked up the answer on a reputable source like NASA? Well, then I would have been able to say, "Ah, okay. So, the answer is Saturn with 146 moons. And in fact, this keeps changing because scientists keep on discovering more and more moons." So I have now grounded my answer in something more believable. I have not hallucinated or made up an answer. Oh, and by the way, I didn't leak personal information about how long ago it's been since I was obsessed with space. Alright, so what does this have to do with large language models? Well, how would a large language model have answered this question? So, let's say that I have a user asking this question about moons. A large language model would confidently say, "Okay, I have been trained, and from what I know in my parameters during my training, the answer is Jupiter." The answer is wrong, but, you know, we don't know. The large language model is very confident in what it answered. Now, what happens when you add this retriever retrieval augmented part here? What does that mean? That means that now, instead of just relying on what the LLM knows, we are adding a content store. This could be open, like the internet. This could be closed, uh, like some collection of documents, collection of policies, whatever. The point though now is that the LLM first goes and talks to the content store and says, "Hey, can you retrieve for me information that is relevant to what the user's query was?" And now, with this retriever augmented answer, it's not Jupiter anymore. We know that it is Saturn. What does this look like? Well, First, user prompts the LLM with the question. They say, this is what my question was. And originally, if we're just talking to a generative model, the generative model says, "Oh, okay, I know the response. Here it is. Here's my response." But now, in the RAG framework, the generative model actually has an instruction that says, "No, no, no. First, go and retrieve relevant content. Combine that with the user's question, and only then generate the answer." So, the prompt now has three parts: the instruction to pay attention to the retrieved content, together with the user's question, now give a response, and in fact, now, you can give evidence for why your response was what it was. So now, hopefully, you can see how does RAG help the two LLM challenges that I had mentioned before. So, first of all, I'll start with the out-of-date part. Now, instead of having to retrain your model, if new information comes up, like, "Hey, we found some more moons," now that's Jupiter again, maybe it'll be Saturn again in the future. All you have to do is you augment your data store with new information, updated information. So now the next time that a user comes and asks the question, we're ready. We just go ahead and retrieve the most up-to-date information. The second problem: source. Well, the large language model is now being instructed to pay attention to primary source data before giving its response, and in fact, now, being able to give evidence. This makes it less likely to hallucinate or to leak data, because it is less likely to rely only on information it learned during training. It also allows us to get the model to uh have a behavior that can be very positive, which is knowing when to say, "I don't know." If the user's question cannot be reliably answered based on your data store, the model should say, "I don't know," instead of making up something that is believable and may mislead the user. This can have a negative effect as well, though, because if the retriever is not sufficiently good to give the large language model the best, most highest-quality grounding information, then maybe the user's query that is answerable doesn't get an answer. So, this is actually why lots of folks, including many of us here at IBM, are working the problem on both sides. We are both working to improve the retriever to give the large language model the best quality data on which to ground its response, and also the generative part, so that the LLM can give the richest, best response, finally, to the user when it generates the answer. Thank you for learning more about RAG, and like and subscribe to the channel. Thank you.

Frequently asked questions

Why is RAG better than just retraining the LLM frequently?

Retraining is expensive and time-consuming. RAG allows instant updates simply by modifying the external content store.

What does it mean for an answer to be "grounded"?

A grounded answer is supported by specific, verifiable evidence retrieved from the external content store. This prevents the model from making up facts.

What are the three parts of a RAG-enhanced prompt?

The prompt includes the user's question, the instruction to pay attention to retrieved content, and the retrieved content itself.

Can RAG leak personal or sensitive information?

RAG reduces the likelihood of leaking data learned during training, but the content store itself must be secured and curated.