Two-Stage Retrieval Pipeline
What you'll learn: How to build a practical retrieve-then-rerank system that balances retrieval quality with speed by configuring candidate set sizes and respecting latency constraints.
The Core Architecture
A two-stage retrieval pipeline separates the search process into two distinct phases:
- First stage (Retrieval): Use a fast bi-encoder to quickly retrieve a large pool of candidate documents from your entire corpus
- Second stage (Reranking): Apply a slower but more accurate cross-encoder to rerank only these candidates
Think of it like a talent show: the first round quickly narrows thousands of applicants to 100 finalists (fast but imprecise), then judges carefully evaluate those 100 to pick the top 10 (slow but accurate).
Configuring the Pipeline
The two key parameters you control:
- Candidate set size (K₁): How many documents the bi-encoder retrieves. Common values: 50-500 depending on your corpus size
- Final set size (K₂): How many reranked results you return. Typically 5-20 for RAG generation
Balancing Quality and Speed
- Larger K₁ = better recall (you won't miss relevant docs), but slower reranking
- Smaller K₁ = faster overall, but risk missing relevant documents that only a cross-encoder would catch
- Your latency budget determines the maximum K₁ you can afford
For example, if reranking takes 20ms per document and you have a 500ms budget, you can rerank at most 25 candidates.
Practical Implementation Pattern
# Stage 1: Fast retrieval (bi-encoder)
candidates = vector_db.search(query_embedding, top_k=100) # K₁
# Stage 2: Precise reranking (cross-encoder)
reranked = cross_encoder.rerank(query, candidates, top_k=10) # K₂
Key Takeaway: Two-stage retrieval lets you search efficiently at scale by using fast bi-encoders for broad recall and precise cross-encoders for final ranking, with tunable candidate sizes that trade quality for speed based on your latency requirements.