Course contentsShow
← AI Engineering
Lesson 334 of 1,8869. Retrieval-Augmented Generation (RAG) - BasicsFree lesson

RAG Limitations and Trade-offs

Recognizing when RAG isn't the solution: retrieval failures, latency costs, and complexity versus simpler approaches.

RAG Limitations and Trade-offs

What you'll learn: When RAG becomes the wrong tool for the job, and the hidden costs you'll pay even when it's right.

The Core Idea

RAG isn't a silver bullet. While it solves many problems (grounding answers in knowledge bases, reducing hallucinations), it introduces complexity, latency, and failure modes that simpler approaches don't have. Sometimes a well-crafted prompt or a fine-tuned model is actually better.

When RAG Falls Short

Retrieval Failures

If your retrieval step returns irrelevant documents (poor embeddings, wrong index configuration, ambiguous queries), the LLM generates answers from bad context. Garbage in, garbage out. You've now built a complex system that performs worse than a simple prompt.

Latency Costs

RAG adds multiple round trips: query embedding → vector search → context assembly → LLM call. Each step adds milliseconds or seconds. For real-time applications, this delay might be unacceptable. A fine-tuned model answers in one hop.

Complexity Overhead

You're now maintaining a knowledge base, embeddings pipeline, vector database, monitoring retrieval quality, and handling index updates. If your knowledge rarely changes and fits in the context window, why not just include it in the prompt directly?

Trade-offs to Consider

RAG shines when:

  • Knowledge changes frequently
  • Context is too large for prompts
  • Transparency and citations matter

Simpler approaches win when:

  • Knowledge is static and small
  • Speed is critical
  • Maintenance resources are limited

Think of RAG like using a library system: powerful for vast collections, but overkill if you only need three books.

Key Takeaway: RAG adds retrieval complexity, latency, and potential failure points. Choose it when dynamic, large-scale knowledge access justifies those costs—not as a default.