Multi-Fidelity Optimization
What you'll learn: How to speed up hyperparameter search by testing promising candidates on easier, cheaper tasks before committing full computational resources.
The Core Idea
Imagine you're a talent scout auditioning 100 singers for a concert. Would you give each person a full 20-minute performance slot immediately? Of course not—you'd do quick 30-second auditions first, eliminate obviously poor fits, then give longer auditions to promising candidates only.
Multi-fidelity optimization applies this same logic to hyperparameter tuning. Instead of training every hyperparameter configuration to completion on your full dataset, you evaluate them using "cheaper approximations" first:
- Smaller datasets (e.g., 10% of training data instead of 100%)
- Fewer training epochs (e.g., 5 epochs instead of 50)
- Lower resolution images (for vision tasks)
- Smaller model architectures (fewer layers/parameters)
These cheaper evaluations are called low-fidelity approximations. Configurations that perform poorly at low fidelity are discarded early. Only the most promising ones graduate to high-fidelity evaluation (full dataset, full epochs).
Why This Works
While low-fidelity results aren't perfectly predictive, they're often "good enough" to identify terrible hyperparameters. You'd rather waste 5 minutes discovering a learning rate is too high than 2 hours training it fully. Methods like Successive Halving and Hyperband (which you've already learned) implement multi-fidelity principles by progressively eliminating poor performers.
Key Takeaway: Multi-fidelity optimization uses cheap approximations—like smaller datasets or fewer epochs—to quickly filter out bad hyperparameters, reserving expensive full evaluations only for promising candidates.