Course contentsShow
← Data Science
Lesson 1418 of 2,14535. Statistical Anomaly DetectionFree lesson

Evaluating Change-Point Detection Methods

Metrics for assessing detection accuracy, false positives, and delay in identifying true change-points.

Evaluating Change-Point Detection Methods

What you'll learn: How to measure whether your change-point detection algorithm is actually finding real changes accurately and quickly.

You've learned several change-point detection methods (CUSUM, PELT, Bayesian approaches), but how do you know if they're working well? Just like evaluating a classification model, you need metrics that capture different aspects of performance.

Core Performance Metrics

Detection Accuracy

The fundamental question: did you find the true change-points? This breaks into:

  • True Positives (TP): Correctly identified change-points
  • False Positives (FP): Flagged changes where none exist (false alarms)
  • False Negatives (FN): Missed actual change-points

From these, you can calculate:

  • Precision: TP / (TP + FP) — of your detections, how many were real?
  • Recall: TP / (TP + FN) — of all real changes, how many did you catch?

Detection Delay

Unlike static anomaly detection, timing matters in change-point detection. Even if you eventually find a change, a 50-point delay might render the detection useless in practice (think: detecting a machine malfunction after it's already caused damage).

Detection delay measures the time lag between when the change actually occurs and when your algorithm flags it. Lower delay is critical for real-time monitoring applications.

Tolerance Windows

Real-world evaluation often uses tolerance: if a detected change-point falls within ±k time steps of the true change, count it as correct. This acknowledges that pinpointing the exact moment of gradual changes is inherently fuzzy.

The Trade-off

Think of it like a smoke detector: too sensitive (low threshold) gives many false alarms but catches fires quickly; too lenient (high threshold) misses real fires or detects them late. You must balance false positive rate against detection delay and recall.

Key Takeaway: Effective evaluation requires tracking precision, recall, false positive rate, and detection delay together—optimizing one metric alone often degrades the others, so choose metrics aligned with your application's priorities.