Anomaly Detection in Practice
You'll learn how to apply anomaly detection to real-world data where you have mostly unlabeled samples and must tune key parameters.
The Real-World Challenge
In practice, you rarely have a perfectly labeled dataset of "normal" and "anomalous" points. Most of your data is unlabeled, with perhaps a tiny fraction confirmed as anomalies. You need to decide: do you treat everything as unlabeled (fully unsupervised) or use your few labeled examples (semi-supervised)?
Key Practical Decisions
Handling Unlabeled Data
Your dataset might contain hidden anomalies even in the "training" set. Algorithms like Isolation Forest and One-Class SVM assume your training data is mostly clean. If you suspect contamination, clean it first or use methods robust to noisy training data.
The Contamination Parameter
Many anomaly detectors ask: "What fraction of the data is anomalous?" This contamination parameter (e.g., contamination=0.05 means 5% are anomalies) guides how many points to flag. If you know your domain (e.g., "1 in 100 transactions is fraud"), set it accordingly. Otherwise, start conservatively (1–5%) and tune based on results.
Unsupervised vs. Semi-Supervised
- Unsupervised: No labels at all. Use Isolation Forest, One-Class SVM, or statistical methods. Good when anomalies are truly unknown.
- Semi-supervised: You have labeled normal data (and maybe a few anomalies). Train One-Class SVM or autoencoders on clean normals, then detect deviations.
Think of it like teaching a guard: unsupervised means "learn what's normal from the crowd," while semi-supervised means "I'll show you examples of normal people first."
Key Takeaway: In practice, anomaly detection requires choosing between unsupervised and semi-supervised approaches, carefully setting contamination levels based on domain knowledge, and ensuring your training data is as clean as possible.