Course contentsShow
← AI Engineering
Lesson 810 of 1,88620. Evaluation Frameworks for LLM SystemsFree lesson

Designing Evaluation Prompts

Writing clear, specific prompts that ask the judge to assess quality, correctness, or other dimensions.

Designing Evaluation Prompts

What you'll learn: How to craft clear, targeted prompts that guide your judge LLM to assess specific quality dimensions reliably.

When using LLM-as-a-Judge, the evaluation prompt is your instruction manual for the judge model. Just like a human grader needs a rubric, your judge LLM needs explicit guidance on what to evaluate and how to score it.

Core Principles

Your evaluation prompt should be specific and focused. Instead of asking "Is this response good?", break it down: "Does this response answer the user's question completely?" or "Is the tone professional and helpful?"

Think of it like giving directions: "Turn left" is vague, but "Turn left at the red mailbox" is actionable. Similarly, "Rate coherence" is vague, but "Does each sentence logically follow the previous one, with clear transitions?" gives the judge concrete criteria.

Essential Components

A strong evaluation prompt includes:

  1. The dimension: What specific aspect to judge (correctness, relevance, safety, tone)
  2. The scale: How to score (1-5, pass/fail, percentage)
  3. Clear criteria: Observable characteristics for each score level
  4. Context: The input, expected behavior, and actual output to evaluate
  5. Output format: How to structure the judgment (score first, then explanation)

Example Structure

Evaluate whether the assistant's response is factually correct.

Input: [user question]
Response: [assistant output]
Reference: [ground truth, if available]

Score 1-5 where:
5 = Completely accurate
3 = Partially correct with minor errors
1 = Factually incorrect

Provide your score, then explain your reasoning.

Key Takeaway: Well-designed evaluation prompts are specific, focused on one dimension at a time, include clear scoring criteria, and request structured output—turning subjective assessment into consistent, actionable feedback.