Course contentsShow
← AI Engineering
Lesson 806 of 1,88620. Evaluation Frameworks for LLM SystemsFree lesson

Metric Selection Strategy

Choosing appropriate metrics based on task type, available ground truth, and downstream business impact requirements.

Metric Selection Strategy

What you'll learn: How to choose the right evaluation metrics for your LLM system based on task type, data availability, and business goals.

Selecting metrics isn't a one-size-fits-all process—it's a strategic decision that shapes how you understand and improve your system. The wrong metrics can make a failing system look successful, or vice versa.

The Three Pillars of Metric Selection

1. Task Type Alignment Your core task dictates baseline metrics. A classification system needs precision and recall. A retrieval system requires ranking metrics like MRR or NDCG. A generation task might need semantic similarity or factuality scores. Start with metrics that directly measure your task's primary objective.

2. Ground Truth Availability What reference data do you have? Gold-standard labels enable precision metrics like accuracy or F1. Human-written ideal outputs allow comparison-based metrics like BLEU or semantic similarity. Limited ground truth? You'll rely more on LLM-as-a-judge or heuristic metrics like instruction following scores.

3. Business Impact Requirements Metrics must connect to real-world outcomes. An e-commerce chatbot might prioritize task completion rate over response fluency. A medical summarization system might weigh factuality detection far above latency. Identify what failure costs your users and business, then choose metrics that catch those failures early.

The Selection Process

Think of it like choosing diagnostic tests: you need metrics that are sensitive (catch real problems), specific (don't cry wolf), and actionable (guide improvements). Combine multiple metrics to form a complete picture—single-metric optimization often backfires.

Key Takeaway: Select metrics by triangulating task requirements, available evaluation data, and downstream business consequences—not by what's easiest to compute or most commonly cited in papers.