Designing Evaluation Prompts
What you'll learn: How to craft clear, targeted prompts that guide your judge LLM to assess specific quality dimensions reliably.
When using LLM-as-a-Judge, the evaluation prompt is your instruction manual for the judge model. Just like a human grader needs a rubric, your judge LLM needs explicit guidance on what to evaluate and how to score it.
Core Principles
Your evaluation prompt should be specific and focused. Instead of asking "Is this response good?", break it down: "Does this response answer the user's question completely?" or "Is the tone professional and helpful?"
Think of it like giving directions: "Turn left" is vague, but "Turn left at the red mailbox" is actionable. Similarly, "Rate coherence" is vague, but "Does each sentence logically follow the previous one, with clear transitions?" gives the judge concrete criteria.
Essential Components
A strong evaluation prompt includes:
- The dimension: What specific aspect to judge (correctness, relevance, safety, tone)
- The scale: How to score (1-5, pass/fail, percentage)
- Clear criteria: Observable characteristics for each score level
- Context: The input, expected behavior, and actual output to evaluate
- Output format: How to structure the judgment (score first, then explanation)
Example Structure
Evaluate whether the assistant's response is factually correct.
Input: [user question]
Response: [assistant output]
Reference: [ground truth, if available]
Score 1-5 where:
5 = Completely accurate
3 = Partially correct with minor errors
1 = Factually incorrect
Provide your score, then explain your reasoning.
Key Takeaway: Well-designed evaluation prompts are specific, focused on one dimension at a time, include clear scoring criteria, and request structured output—turning subjective assessment into consistent, actionable feedback.