Course contentsShow
← AI Engineering
Lesson 396 of 1,88610. RAG - Improving RetrievalFree lesson

Two-Stage Retrieval Pipelines

Combining fast vector search with slower reranking for optimal speed-accuracy tradeoffs.

Two-Stage Retrieval Pipelines

What You'll Learn

You'll learn how to combine fast vector search with slower reranking to get the best of both worlds: speed and accuracy.

The Core Idea

Think of hiring for a job: you first filter 1,000 resumes down to 50 candidates using quick criteria (years of experience, keywords), then spend quality time interviewing those 50 to find the best 5.

A two-stage retrieval pipeline works the same way:

  1. Stage 1 (Fast Retrieval): Use vector search to quickly retrieve a large candidate set (e.g., top 50-100 chunks) from your knowledge base. This is fast because vector databases are optimized for speed using approximate nearest neighbor algorithms.

  2. Stage 2 (Precise Reranking): Use a cross-encoder reranking model to carefully score those candidates and select the top-k most relevant (e.g., top 5). This is slower per document but runs on a much smaller set.

Why This Trade-off Works

Vector search excels at recall (finding potentially relevant documents) but may not perfectly rank them. Cross-encoders excel at precision (identifying the truly best matches) but are too slow to run on thousands of documents.

By separating these concerns, you get:

  • ⚡ Speed: Vector search handles the bulk filtering efficiently
  • 🎯 Accuracy: Reranking fine-tunes the final selection
  • 💰 Cost-effectiveness: Only rerank what's likely relevant

For example, searching 100,000 chunks takes milliseconds with vectors, then reranking 50 candidates takes a few hundred milliseconds—still fast enough for real-time RAG applications.

Key Takeaway: Two-stage pipelines balance speed and accuracy by using fast vector search for broad retrieval and slower cross-encoder reranking for precise final selection.