Top-K Retrieval and Similarity Metrics
What you'll learn: How retrieval systems use similarity metrics to rank and select the most relevant document chunks for your query.
The Ranking Challenge
Once you have embedded both your query and a large collection of document chunks into the same vector space (using bi-encoders), you need to rank them by relevance. You don't want all thousands of chunks—just the best ones. This is where similarity metrics and top-K selection come in.
Similarity Metrics: Measuring Closeness
The most common metric in dense retrieval is cosine similarity, which measures the angle between two vectors. It ranges from -1 (opposite) to 1 (identical direction), with higher scores meaning more similar content. Cosine similarity focuses on direction, not magnitude—perfect for comparing semantic meaning.
Other metrics include:
- Dot product: Similar to cosine but sensitive to vector magnitude
- Euclidean distance: Measures straight-line distance between points (lower = more similar)
Think of it like a music recommendation system: cosine similarity asks "do these songs point in the same stylistic direction?" rather than "how loud are they?"
Top-K Selection
After computing similarity scores between your query embedding and every document chunk embedding, you sort the chunks by score and take the top K results—typically K=3 to K=10 for RAG systems.
Why not take everything? Token limits, cost, and noise. Irrelevant context confuses the LLM during generation. Top-K gives you the sweet spot: enough context to be comprehensive, but focused enough to be useful.
The choice of K depends on your task: complex questions might need K=10, while simple lookups work with K=3.
Key Takeaway: Similarity metrics like cosine similarity rank document relevance to your query, and top-K retrieval selects only the most relevant chunks to pass to your LLM, balancing context richness with focus.