Multi-Dimensional Scoring
What you'll learn: How to combine multiple evaluation metrics into a single composite score that reflects overall LLM system quality across different dimensions.
The Challenge of Multiple Metrics
You've learned individual metrics like classification accuracy, retrieval precision, semantic similarity, factuality scores, and latency. But real-world LLM systems need to excel across all these dimensions simultaneously. How do you make a single judgment call when one model is faster but less accurate, and another is more factual but slower?
The Core Idea
Multi-dimensional scoring creates a composite score by combining multiple metrics with weights that reflect their relative importance to your use case. Think of it like a restaurant review: you might weight food quality at 50%, service at 30%, and ambiance at 20% to get an overall rating.
Basic Approaches
Weighted Average:
composite_score = (0.4 × accuracy) + (0.3 × relevance) + (0.2 × coherence) + (0.1 × speed_score)
Normalized Metrics: First normalize each metric to a 0-1 scale, then combine them. This prevents a metric like latency (measured in milliseconds) from overwhelming a 0-1 accuracy score.
Threshold-Based: Some dimensions might be requirements rather than trade-offs. For example: only consider models with latency < 2s, then score on quality metrics.
Practical Considerations
- Domain alignment: Customer support might prioritize factuality (0.5) and tone (0.3) over speed (0.2)
- Stakeholder input: Business teams help define weights
- Iterative tuning: Adjust weights based on real-world performance
- Transparency: Always report individual metrics alongside the composite score for debugging
Key Takeaway: Multi-dimensional scoring combines metrics like accuracy, relevance, factuality, and latency into a single weighted composite score that reflects your system's priorities and enables clear model comparisons.