Course contentsShow
← Data Science
Lesson 1378 of 2,14535. Statistical Anomaly DetectionFree lesson

Setting Z-Score Thresholds

Choosing appropriate cutoff values (typically ±2, ±3) based on the normal distribution and false positive rates.

Setting Z-Score Thresholds

You'll learn: How to choose the right cutoff values for flagging anomalies using Z-scores, balancing detection sensitivity with false alarm rates.

The Core Idea

Once you've calculated Z-scores for your data, you need to decide: how extreme does a value need to be before you call it an anomaly? This is where threshold selection comes in. The most common thresholds are ±2 and ±3 standard deviations from the mean, and your choice directly affects your false positive rate.

Think of it like a security alarm: set it too sensitive (low threshold like ±2) and you'll catch more real threats but also get more false alarms. Set it less sensitive (high threshold like ±3) and you'll miss some genuine issues but have fewer false positives.

Understanding the Numbers

In a normal distribution:

  • ±2 standard deviations captures about 95% of data → ~5% flagged as anomalies
  • ±3 standard deviations captures about 99.7% of data → ~0.3% flagged as anomalies
import numpy as np

z_scores = (data - data.mean()) / data.std()

# Conservative threshold (fewer false positives)
anomalies_3sigma = np.abs(z_scores) > 3

# More sensitive threshold (catches more anomalies)
anomalies_2sigma = np.abs(z_scores) > 2

Choosing Your Threshold

Your choice depends on context:

  • Use ±3 when false positives are costly (e.g., shutting down production systems)
  • Use ±2 when missing real anomalies is costly (e.g., fraud detection)
  • Use custom values (like ±2.5) to fine-tune based on domain knowledge and historical patterns

Key Takeaway: Z-score thresholds balance sensitivity and specificity—choose ±2 for broader detection with more false positives, or ±3 for stricter detection with fewer false alarms, always considering your specific use case and the cost of errors.