Metric Selection Strategy
What you'll learn: How to choose the right evaluation metrics for your LLM system based on task type, data availability, and business goals.
Selecting metrics isn't a one-size-fits-all process—it's a strategic decision that shapes how you understand and improve your system. The wrong metrics can make a failing system look successful, or vice versa.
The Three Pillars of Metric Selection
1. Task Type Alignment Your core task dictates baseline metrics. A classification system needs precision and recall. A retrieval system requires ranking metrics like MRR or NDCG. A generation task might need semantic similarity or factuality scores. Start with metrics that directly measure your task's primary objective.
2. Ground Truth Availability What reference data do you have? Gold-standard labels enable precision metrics like accuracy or F1. Human-written ideal outputs allow comparison-based metrics like BLEU or semantic similarity. Limited ground truth? You'll rely more on LLM-as-a-judge or heuristic metrics like instruction following scores.
3. Business Impact Requirements Metrics must connect to real-world outcomes. An e-commerce chatbot might prioritize task completion rate over response fluency. A medical summarization system might weigh factuality detection far above latency. Identify what failure costs your users and business, then choose metrics that catch those failures early.
The Selection Process
Think of it like choosing diagnostic tests: you need metrics that are sensitive (catch real problems), specific (don't cry wolf), and actionable (guide improvements). Combine multiple metrics to form a complete picture—single-metric optimization often backfires.
Key Takeaway: Select metrics by triangulating task requirements, available evaluation data, and downstream business consequences—not by what's easiest to compute or most commonly cited in papers.