Adding Metadata Filtering
What you'll learn: How to narrow semantic search results by filtering on metadata fields like dates, authors, or document types before or after embedding-based retrieval.
Imagine searching a library for books about "climate change" — but you only want scientific papers published after 2020, not newspaper articles. Semantic search finds documents by meaning, but metadata filtering lets you add precise constraints based on structured information.
Core Concept
When you store document embeddings, you typically also store metadata alongside each vector — things like:
date: publication dateauthor: document creatordoc_type: "article", "report", "email"category: topic labelslanguage: document language
Post-retrieval filtering works like this:
- Perform semantic search to get top-K most similar documents (e.g., top 100)
- Filter those results based on metadata conditions
- Return the filtered subset to the user
This approach keeps your search semantically relevant while respecting hard constraints.
Example Flow
# After computing cosine similarity and retrieving top_results
filtered_results = [
doc for doc in top_results
if doc['metadata']['date'] >= '2020-01-01'
and doc['metadata']['doc_type'] == 'research_paper'
]
You've combined the power of embeddings (semantic matching) with traditional filtering (exact metadata matching).
When to Use It
- Time-bound queries: "recent reports on AI safety"
- Author-specific searches: "papers by Zhang"
- Document type restrictions: "only email threads"
- Multi-tenant systems: filtering by user permissions
Key Takeaway: Metadata filtering adds precise constraints to semantic search, letting you combine meaning-based retrieval with structured filters for more targeted results.