Dense vs Sparse Retrieval
What you'll learn: How keyword-based sparse retrieval differs from embedding-based dense retrieval for finding relevant documents.
The Core Difference
When you need to find relevant documents from a large collection (like answering "What is photosynthesis?" from Wikipedia), you can use two fundamentally different approaches:
Sparse retrieval methods like BM25 and TF-IDF work by matching exact keywords. They represent documents as sparse vectors where most entries are zero, with non-zero values only for words that actually appear. If your query is "machine learning," these methods look for documents containing those exact words (or very close variants).
Dense retrieval uses neural networks to create embedding vectors where semantically similar texts have similar representations, even without shared keywords. Your query and documents are encoded into dense vectors (all entries have values), then similarity is measured by distance in this embedding space.
Why It Matters
Sparse methods excel at exact matching and are computationally efficient, but they miss semantic connections. Searching for "ML algorithms" won't match "machine learning techniques" well with pure keyword matching.
Dense retrieval captures semantic meaning—it understands that "canine" and "dog" are related without exact word overlap. This enables true semantic search where meaning matters more than exact phrasing.
The Trade-off
Sparse retrieval is fast, interpretable (you can see which keywords matched), and works well for precise terminology. Dense retrieval handles paraphrasing and conceptual similarity better but requires trained neural encoders and more compute for encoding.
Modern systems often combine both: sparse retrieval for efficient initial filtering, dense retrieval for semantic reranking.
Key Takeaway: Sparse retrieval (BM25, TF-IDF) matches exact keywords efficiently, while dense retrieval uses neural embeddings to understand semantic similarity beyond word overlap.