Evaluating Change-Point Detection Methods
What you'll learn: How to measure whether your change-point detection algorithm is actually finding real changes accurately and quickly.
You've learned several change-point detection methods (CUSUM, PELT, Bayesian approaches), but how do you know if they're working well? Just like evaluating a classification model, you need metrics that capture different aspects of performance.
Core Performance Metrics
Detection Accuracy
The fundamental question: did you find the true change-points? This breaks into:
- True Positives (TP): Correctly identified change-points
- False Positives (FP): Flagged changes where none exist (false alarms)
- False Negatives (FN): Missed actual change-points
From these, you can calculate:
- Precision: TP / (TP + FP) — of your detections, how many were real?
- Recall: TP / (TP + FN) — of all real changes, how many did you catch?
Detection Delay
Unlike static anomaly detection, timing matters in change-point detection. Even if you eventually find a change, a 50-point delay might render the detection useless in practice (think: detecting a machine malfunction after it's already caused damage).
Detection delay measures the time lag between when the change actually occurs and when your algorithm flags it. Lower delay is critical for real-time monitoring applications.
Tolerance Windows
Real-world evaluation often uses tolerance: if a detected change-point falls within ±k time steps of the true change, count it as correct. This acknowledges that pinpointing the exact moment of gradual changes is inherently fuzzy.
The Trade-off
Think of it like a smoke detector: too sensitive (low threshold) gives many false alarms but catches fires quickly; too lenient (high threshold) misses real fires or detects them late. You must balance false positive rate against detection delay and recall.
Key Takeaway: Effective evaluation requires tracking precision, recall, false positive rate, and detection delay together—optimizing one metric alone often degrades the others, so choose metrics aligned with your application's priorities.