Cross-Encoders for Reranking
What you'll learn: How cross-encoders jointly process query-document pairs to dramatically improve retrieval accuracy as a second-stage reranking step.
The Core Idea
In multi-stage retrieval, bi-encoders (like Sentence Transformers) quickly retrieve candidate documents by comparing pre-computed embeddings. But they encode query and document separately, missing rich interaction signals. Cross-encoders solve this by processing the query and document together in a single forward pass, enabling the model to capture nuanced relevance patterns.
Think of it like dating: a bi-encoder creates separate profiles and matches based on compatibility scores computed independently. A cross-encoder is like an actual date—the two people interact directly, revealing chemistry that profiles alone can't capture.
How Cross-Encoders Work
Instead of producing separate embeddings, a cross-encoder takes the concatenated [query, document] pair as input and outputs a single relevance score. The transformer's attention mechanism can model token-level interactions between query and document, capturing:
- Exact phrase matches in context
- Semantic relationships between query terms and document passages
- Negations, qualifiers, and subtle meaning shifts
This joint encoding is computationally expensive—you must run inference for every query-document pair. That's why cross-encoders work best as a reranker: use a fast bi-encoder to retrieve the top 100-1000 candidates, then rerank the top 20-50 with a cross-encoder for maximum accuracy.
The Trade-off
Cross-encoders deliver superior accuracy but cannot pre-compute document representations. For every query, you need fresh inference on all candidate pairs. Bi-encoders are fast but less precise; cross-encoders are slow but highly accurate.
Key Takeaway: Cross-encoders jointly encode query-document pairs for fine-grained relevance scoring, making them ideal second-stage rerankers after fast bi-encoder retrieval—accuracy at the cost of computational efficiency.