Course contentsShow
← Machine Learning and Deep Learning
Lesson 3017 of 3,53866. Production Monitoring and Data DriftFree lesson

Online vs Offline Metrics: The Feedback Loop Challenge

Understanding the delay between predictions and ground truth labels and strategies for monitoring without immediate feedback.

Online vs Offline Metrics: The Feedback Loop Challenge

What you'll learn: Why measuring model performance in production is fundamentally different from evaluation during development, and how to monitor when ground truth arrives slowly or never.

The Core Problem

In training and testing, you have immediate access to labels: you predict, compare to ground truth, and calculate metrics instantly. In production, there's often a feedback loop delay—the time between making a prediction and learning whether it was correct.

Real-World Examples

  • Recommendation systems: You predict a user will click an item now, but learn if they actually clicked only after they've browsed the page (seconds to minutes delay)
  • Fraud detection: You flag a transaction as suspicious today, but confirm fraud only after investigation completes (days to weeks)
  • Loan default prediction: You approve a loan, but learn the outcome months or years later
  • Medical diagnosis: Predictions made today, outcomes known after treatment completion

The Monitoring Challenge

Offline metrics (accuracy, F1, AUC) require ground truth labels. Online metrics must work without immediate labels:

  • Proxy metrics: Click-through rate, engagement time, conversion rate
  • Model health indicators: Prediction confidence distribution, feature statistics
  • Business metrics: Revenue per user, complaint rate, manual review volume

Think of it like navigating by compass vs. landmarks: offline metrics are your landmarks (precise, but you need to reach them first), while online metrics are your compass (immediate, but indirect).

Monitoring Strategies

  1. Log predictions with timestamps to join with delayed labels later
  2. Sample a subset for manual labeling to get faster feedback
  3. Monitor proxy signals that correlate with true outcomes
  4. Track prediction distribution shifts as early warning signs

Key Takeaway: Production monitoring requires designing online metrics that don't need immediate ground truth, while building infrastructure to retroactively evaluate offline metrics once feedback arrives—often combining both to balance speed and accuracy.