Two-Stage Retrieval Pipelines
What You'll Learn
You'll learn how to combine fast vector search with slower reranking to get the best of both worlds: speed and accuracy.
The Core Idea
Think of hiring for a job: you first filter 1,000 resumes down to 50 candidates using quick criteria (years of experience, keywords), then spend quality time interviewing those 50 to find the best 5.
A two-stage retrieval pipeline works the same way:
-
Stage 1 (Fast Retrieval): Use vector search to quickly retrieve a large candidate set (e.g., top 50-100 chunks) from your knowledge base. This is fast because vector databases are optimized for speed using approximate nearest neighbor algorithms.
-
Stage 2 (Precise Reranking): Use a cross-encoder reranking model to carefully score those candidates and select the top-k most relevant (e.g., top 5). This is slower per document but runs on a much smaller set.
Why This Trade-off Works
Vector search excels at recall (finding potentially relevant documents) but may not perfectly rank them. Cross-encoders excel at precision (identifying the truly best matches) but are too slow to run on thousands of documents.
By separating these concerns, you get:
- ⚡ Speed: Vector search handles the bulk filtering efficiently
- 🎯 Accuracy: Reranking fine-tunes the final selection
- 💰 Cost-effectiveness: Only rerank what's likely relevant
For example, searching 100,000 chunks takes milliseconds with vectors, then reranking 50 candidates takes a few hundred milliseconds—still fast enough for real-time RAG applications.
Key Takeaway: Two-stage pipelines balance speed and accuracy by using fast vector search for broad retrieval and slower cross-encoder reranking for precise final selection.