Course contentsShow
← AI Engineering
Lesson 796 of 1,88620. Evaluation Frameworks for LLM SystemsFree lesson

Classification Task Metrics

Measuring accuracy, precision, recall, and F1-score for LLM outputs that map to predefined categories or labels.

Classification Task Metrics

What you'll learn: How to measure the quality of LLM outputs that assign labels or categories using accuracy, precision, recall, and F1-score.

Why Classification Metrics Matter

When your LLM classifies text—like determining sentiment (positive/negative), categorizing support tickets, or labeling intents—you need quantitative ways to evaluate performance beyond "it looks good." Classification metrics give you precise measurements of how well your system maps inputs to predefined categories.

The Four Core Metrics

Accuracy

The percentage of predictions that are correct overall. If your LLM correctly labeled 85 out of 100 support tickets, that's 85% accuracy. Simple, but can be misleading when categories are imbalanced (e.g., 95% of tickets are "billing" queries).

Precision

Of all the items your LLM labeled as a specific category, what fraction were actually correct? High precision means few false positives. Think of it as "when the model says X, how often is it right?"

Recall

Of all the items that truly belong to a category, what fraction did your LLM correctly identify? High recall means few false negatives. It answers: "did we catch all the important cases?"

F1-Score

The harmonic mean of precision and recall, giving you a single balanced metric. Useful when you care equally about catching all instances (recall) and not mislabeling things (precision). Ranges from 0 (worst) to 1 (perfect).

When to Use Each

  • Accuracy: Quick overview when classes are balanced
  • Precision: When false positives are costly (e.g., spam filtering)
  • Recall: When missing cases is costly (e.g., detecting urgent support issues)
  • F1: When you need balance or to compare models holistically

Key Takeaway: Classification metrics translate LLM categorization performance into numbers—accuracy shows overall correctness, precision measures false positive control, recall captures completeness, and F1 balances both for robust evaluation.