Bi-Encoders vs Cross-Encoders
What you'll learn: You'll understand when to use bi-encoders versus cross-encoders for semantic search and text similarity tasks, and the crucial speed-versus-accuracy trade-off between them.
The Core Difference
Imagine you're comparing two documents. A bi-encoder processes each document independently through separate (or shared) neural networks, producing fixed embeddings. You then compare these embeddings using simple operations like cosine similarity. A cross-encoder, on the other hand, concatenates both documents and feeds them together through a single network that directly outputs a similarity score.
Bi-Encoders: Fast But Independent
Bi-encoders are the foundation of Sentence Transformers. They encode your query once and your document collection once. Because documents are encoded independently, you can:
- Pre-compute and store all document embeddings
- Compare millions of documents quickly using vector similarity
- Scale efficiently for large-scale retrieval
Limitation: The query and document never "see" each other during encoding, missing subtle interactions.
Cross-Encoders: Accurate But Slow
Cross-encoders process query and document jointly, allowing full attention between every word pair. This captures nuanced relationships bi-encoders miss.
Limitation: You cannot pre-compute embeddings. Each query-document pair requires a fresh forward pass through the entire network—too slow for searching millions of documents.
The Practical Workflow
Most production systems use both: bi-encoders retrieve the top-k candidates quickly (e.g., top 100 from millions), then cross-encoders re-rank those candidates for maximum accuracy.
Key Takeaway: Bi-encoders excel at fast, large-scale retrieval by encoding independently; cross-encoders provide superior accuracy through joint encoding but are too slow for initial search—use bi-encoders first, then re-rank with cross-encoders.