RAG-Specific Prompt Structure
What you'll learn: How to architect prompts that clearly separate retrieved context, user queries, and instructions to maximize RAG system performance.
Why Structure Matters
Unlike standard prompting, RAG prompts must handle three distinct elements: the user's original question, the retrieved context from your vector database, and instructions for how to use that context. Poor organization leads to confused models that ignore context, hallucinate, or fail to cite sources properly.
Think of it like a courtroom: the judge (LLM) needs the case facts (retrieved context) clearly separate from the question being asked (user query) and the rules for how to decide (instructions). Mix them together, and you get chaos.
The Three-Part Pattern
1. System Instructions (Top) Set the behavior rules first: "You are an assistant that answers questions using only the provided context. Always cite sources."
2. Retrieved Context (Middle) Clearly labeled and formatted:
Context:
---
[Document 1 - Source: report.pdf]
The Q4 revenue was $2.1M...
[Document 2 - Source: memo.txt]
The forecast shows...
---
3. User Query (Bottom) "Based on the context above, what was Q4 revenue?"
Why This Order Works
- Instructions first prime the model's behavior before it sees any content
- Context before query prevents the model from generating answers before reading evidence
- Clear boundaries (like
---markers) help the model distinguish sections - Metadata inclusion enables proper citation generation
This structure directly addresses context window constraints by making efficient use of available tokens and reducing the lost-in-the-middle problem you learned about earlier.
Key Takeaway: Structure RAG prompts with explicit sections—instructions, then context with metadata, then query—to ensure the LLM processes information in the right order and uses retrieved knowledge correctly.