Course contentsShow
← Machine Learning and Deep Learning
Lesson 2030 of 3,53843. Retrieval-Augmented Generation: Advanced TechniquesFree lesson

Evaluating Semantic Similarity vs Task Relevance

Distinguishing between documents that are semantically similar and those that actually help answer the query.

Evaluating Semantic Similarity vs Task Relevance

What you'll learn: You'll understand why a document can sound related to your query but still fail to help answer it—and why this distinction matters for RAG evaluation.

The Core Problem

When evaluating retrieval systems, there's a subtle but critical difference between semantic similarity and task relevance. Semantic similarity measures how closely two pieces of text relate to similar topics or concepts. Task relevance measures whether a document actually helps accomplish the user's goal.

The Disconnect

Imagine a user asks: "How do I fix a broken HTTP connection in Python?"

  • High semantic similarity, low task relevance: A document explaining the HTTP protocol's history and architecture scores high on similarity (lots of shared vocabulary: "HTTP," "connection," "protocol") but provides zero actionable help.

  • Lower semantic similarity, high task relevance: A troubleshooting guide titled "Common Socket Errors" might use different terminology but contains the exact solution for resetting connections.

This is why NDCG and MRR metrics (which you've learned) incorporate graded relevance judgments, not just binary "similar/not similar" decisions.

Why This Matters for RAG

Your embedding model optimizes for semantic similarity—it groups conceptually related text. But your user needs task relevance. A RAG system retrieving highly similar but unhelpful documents wastes context window tokens and degrades generation quality.

Evaluation Strategy

When creating ground truth (lesson 2029), annotators must ask: "Would this document help the LLM answer the question?" rather than "Does this document discuss the same topic?" This task-oriented labeling ensures your metrics reflect real-world usefulness.

Key Takeaway: Semantic similarity captures topical overlap; task relevance captures utility for answering the query. Effective RAG evaluation requires measuring the latter, even though retrieval systems optimize for the former.