Why Reranking Matters in RAG
What you'll learn: Why pure vector similarity isn't always enough and how reranking can fix retrieval quality issues.
The Vector Similarity Limitation
When you retrieve documents using embeddings, you're relying on vector similarity (like cosine distance) to determine relevance. This works well for semantic matching, but has blind spots:
- Semantic similarity ≠ actual relevance: Two documents might be semantically similar to your query but differ drastically in usefulness
- No query-specific ranking: Vector search doesn't understand why you're asking or what makes one result better than another for your specific use case
- Missing nuance: Embeddings compress meaning into fixed-size vectors, losing fine-grained details like factual accuracy, recency, or authority
Think of it like a librarian who only matches books by their cover descriptions. They might hand you three mystery novels when you asked for "solving a puzzle," but one is actually a children's book, one is academic, and one is the perfect detective story you wanted.
What Reranking Does
Reranking takes your initially retrieved results (say, top 20 candidates) and re-scores them using a more sophisticated model. This second-pass model:
- Understands the relationship between query and document more deeply
- Can consider factors embeddings miss (keyword overlap, document structure, answer likelihood)
- Typically uses cross-encoders or LLMs that process query + document together
The result? Your top 3-5 documents shown to the LLM are much more likely to contain the actual answer, improving generation quality while keeping retrieval costs reasonable.
Key Takeaway: Reranking acts as a quality filter after vector retrieval, using more compute to deeply evaluate query-document relevance and surface the truly best matches for generation.