Handling Empty Retrievals
What you'll learn: How to gracefully manage situations when your retrieval system finds nothing relevant to answer the user's question.
The Problem
Even with a well-designed RAG system, retrieval can fail. Your vector search might return zero results, or all returned chunks might fall below your relevance threshold. This happens when:
- The user asks about information not in your knowledge base
- The query is too vague or uses unfamiliar terminology
- Your similarity threshold is set too high
Ignoring this scenario leads to confusing or hallucinated responses.
Why It Matters
When retrieval comes up empty, your LLM still needs to generate something. Without explicit instructions, the model might:
- Fabricate an answer using its training data (hallucination)
- Generate a generic, unhelpful response
- Pretend it found relevant context when it didn't
The Solution
Be explicit in your prompt template. Include instructions for the empty-retrieval case:
Context: {context}
If the context above is empty or doesn't contain relevant information,
politely inform the user that you don't have information about their
question in your knowledge base. Do not make up answers.
Question: {question}
You can also implement upstream handling by detecting empty retrievals before calling the LLM and returning a pre-crafted message:
if not retrieved_docs or all(doc.score < threshold):
return "I don't have information about that in my knowledge base."
This approach saves API costs and prevents hallucination.
Key Takeaway: Always handle empty retrievals explicitly—either through prompt instructions or pre-generation checks—to maintain honesty and avoid hallucinations when your knowledge base doesn't contain the answer.