Course contentsShow
← AI Engineering
Lesson 1233 of 1,88630. Monitoring and ObservabilityFree lesson

Quality Metrics for Generations

Measure output quality through completion rates, format validity, hallucination scores, and relevance indicators.

Quality Metrics for Generations

You'll learn how to systematically measure the quality of LLM outputs using four key dimensions: completion, format, hallucination, and relevance.

The Core Challenge

When your LLM system is running in production, latency and cost aren't enough—you need to know if the outputs are good. Quality metrics help you detect when your model starts producing incomplete responses, invalid JSON, fabricated facts, or off-topic content. Think of these as health checks for your AI's brain.

The Four Pillars of Generation Quality

1. Completion Rates

Track how often the model finishes its response versus hitting token limits or stopping mid-sentence. A sudden drop from 99% to 85% completion might signal context window issues or prompt problems.

2. Format Validity

When you expect structured output (JSON, XML, markdown tables), measure parsing success rates. If your system expects {"name": "...", "age": 25} but gets malformed JSON 15% of the time, your downstream pipeline breaks.

3. Hallucination Scores

Detect when the model invents facts. This might involve checking citations against source documents, comparing outputs to ground truth, or using consistency checks across multiple generations. High hallucination rates = trust erosion.

4. Relevance Indicators

Measure how well outputs match the user's intent. This could be as simple as keyword matching for search queries or as sophisticated as embedding similarity between the question and answer. Off-topic responses waste user time and tokens.

Why This Matters Now

You've already learned request-level instrumentation (lesson 1232). Now you're adding semantic instrumentation—measuring what the content actually says, not just how fast it arrived.

Key Takeaway: Quality metrics transform black-box LLM outputs into measurable, monitorable signals that catch regressions before users complain.