Cross-Encoder Models for Reranking
What you'll learn: How cross-encoder models work and why they deliver superior accuracy compared to bi-encoders when scoring relevance between queries and documents.
How Bi-Encoders vs Cross-Encoders Differ
In earlier lessons, you've used bi-encoders (embedding models) that encode queries and documents separately into vector space. They're fast because documents are pre-encoded, and retrieval is just nearest-neighbor search.
Cross-encoders take a fundamentally different approach: they process the query and each candidate document together as a single input pair. Think of it like this:
- Bi-encoder: "Does this apple look like this orange?" (comparing two separate photos)
- Cross-encoder: "How similar are this apple and orange when I look at them side-by-side?" (examining both together)
How Cross-Encoders Work
A cross-encoder concatenates your query and a candidate document, feeds them into a transformer model (like BERT), and outputs a single relevance score. The attention mechanism sees both texts simultaneously, allowing it to understand fine-grained relationships between query terms and document content.
Why this matters for reranking: After your initial retrieval (using fast bi-encoders) returns, say, 100 candidates, you can pass the top 20 through a cross-encoder. The cross-encoder rescores them with much higher precision, promoting truly relevant results to the top.
The Trade-off
Cross-encoders are more accurate because they see query-document interactions directly. But they're slower — you can't pre-compute document representations. That's why they're perfect for reranking a small set of candidates, not searching millions of documents.
Key Takeaway: Cross-encoders rerank by jointly processing query-document pairs for precise relevance scoring, delivering higher accuracy than bi-encoders at the cost of speed — making them ideal as a second-stage reranker in RAG pipelines.