Multi-Aspect Evaluation with LLM Judges
What you'll learn: How to break down complex LLM output quality into separate evaluable dimensions and combine them for more nuanced judgments.
The Problem with Single Scores
When using an LLM-as-Judge to evaluate generated responses, asking for one overall quality score oversimplifies the task. A response might be helpful but factually wrong, or truthful but poorly formatted. A single number can't capture these trade-offs.
Breaking Down the Evaluation
Multi-aspect evaluation means judging outputs across separate, well-defined dimensions:
- Helpfulness: Does it address the user's request?
- Harmlessness: Is it safe, non-toxic, and appropriate?
- Factuality: Are claims accurate and verifiable?
- Style: Is it well-written, clear, and properly formatted?
- Coherence: Does it flow logically?
Each dimension gets its own score (e.g., 1-5 scale) and reasoning. This mirrors how human annotators evaluate complex tasks—one aspect at a time.
Why This Matters
Separate dimensions let you:
- Diagnose weaknesses: Maybe your model is helpful but often inaccurate
- Weight differently: In medical applications, factuality might matter more than style
- Reduce bias: Judges reasoning about one clear criterion are less likely to conflate issues
Aggregating Scores
After scoring each aspect, you can:
- Take a weighted average (e.g., 2× factuality + 1× style)
- Use minimum score (worst aspect defines quality)
- Report a profile instead of one number
- Apply task-specific rules (e.g., helpfulness only counts if harmless)
Think of it like a restaurant review: food, service, ambiance, and value each matter separately, but you might weight them differently depending on the occasion.
Key Takeaway: Multi-aspect evaluation decomposes complex judgments into separate dimensions (helpfulness, harmlessness, factuality, style), enabling more precise diagnosis and flexible aggregation based on your priorities.