Course contentsShow
← Machine Learning and Deep Learning
Lesson 1334 of 3,53829. Core NLP TasksFree lesson

Late Interaction Models (ColBERT)

Encode queries and documents separately but perform fine-grained token-level interactions at retrieval time.

Late Interaction Models (ColBERT)

What you'll learn: How ColBERT achieves the efficiency of bi-encoders while preserving the accuracy of cross-encoders through delayed token-level matching.

The Hybrid Approach

So far, you've learned that bi-encoders encode queries and documents separately (fast but less accurate), while cross-encoders process them together (accurate but slow). ColBERT (Contextualized Late Interaction over BERT) cleverly splits the difference.

The key innovation: encode queries and documents independently like bi-encoders, but instead of producing a single vector per text, preserve all token embeddings. Then at retrieval time, perform fine-grained interactions between query tokens and document tokens.

How It Works

Think of it like a restaurant menu search:

  • Bi-encoder: Represents the entire menu as one summary vector. Fast, but loses details.
  • Cross-encoder: Reads your question alongside each menu item together. Accurate, but must re-read everything for each query.
  • ColBERT: Pre-processes each menu item into detailed ingredient-level descriptions. When you ask a question, it quickly compares your query words against those detailed descriptions without re-reading the full menu.

ColBERT computes a MaxSim operation: for each query token, find its maximum similarity with any document token, then sum these scores. This captures which query terms match which document parts.

The Benefits

You get document representations you can pre-compute and index (like bi-encoders), but with token-level precision that approaches cross-encoder quality. The retrieval is slower than pure bi-encoders but much faster than cross-encoders, making it practical for large-scale search.

Key Takeaway: ColBERT bridges bi-encoder efficiency and cross-encoder accuracy by encoding texts separately but deferring fine-grained token interactions until retrieval time, achieving a sweet spot for semantic search at scale.