Diagnosing Retrieval Failures
What you'll learn: How to systematically debug when your RAG system returns no results or irrelevant documents by inspecting queries, embeddings, and index state.
When your RAG system fails to retrieve useful documents (or returns nothing at all), you need detective skills. You've already learned about common failure modes—now it's time to diagnose why they're happening so you can fix them.
The Diagnostic Process
Think of retrieval failure diagnosis like troubleshooting a broken search engine. You need to check three critical components:
1. Query Inspection
Start at the source. Print the actual query being sent to your vector store. Is it what you expected? Sometimes query optimization or preprocessing mangles the user's intent. Check if stopwords were removed too aggressively, or if query expansion added confusing terms.
2. Embedding Analysis
Your query gets converted to a vector embedding. Does it make sense? Compare your query embedding to known document embeddings using cosine similarity manually. If semantically similar phrases produce wildly different embeddings, your embedding model might not suit your domain (medical queries with a general-purpose model, for example).
3. Index State Verification
Is your vector index actually populated? Check document count, sample a few stored vectors, and verify metadata fields exist. An empty index or corrupted data explains zero results immediately. Also check if filtering conditions (metadata-based pre-filtering) are too restrictive.
Practical Debugging Steps
- Log the raw query, processed query, and top-K similarity scores
- Manually query your index with known-good test cases
- Check if distance thresholds are rejecting all candidates
- Verify your embedding model matches what was used during indexing
- Inspect a few retrieved documents' actual text content—not just scores
Key Takeaway: Retrieval failure diagnosis requires inspecting the full pipeline—from query formulation through embedding generation to index lookup—using logging, manual similarity checks, and index state verification to isolate where the breakdown occurs.