Performance Estimation Strategies in NAS
What you'll learn: How to speed up Neural Architecture Search by avoiding full training for every candidate architecture.
The Core Problem
In Neural Architecture Search, you need to compare potentially thousands of architectures. Training each one from scratch to convergence would take months or years of compute time. Performance estimation strategies let you predict which architectures are promising without wasting resources on full training.
Three Key Acceleration Techniques
Early Stopping
Train each candidate for just a few epochs instead of full convergence. Architectures that perform poorly early are likely to remain poor performers—so stop them and move on. Think of it like a cooking competition: a chef struggling with basic knife skills in the first 5 minutes probably won't win the full hour-long challenge.
Weight Inheritance
Instead of random initialization, inherit weights from a previously evaluated similar architecture. If you've just tested a 5-layer network and now want to test a 6-layer variant, copy the first 5 layers' weights. This gives the new architecture a "warm start" and reaches meaningful performance faster.
Proxy Tasks
Train on simplified versions of your target task: fewer training samples, lower resolution images, or fewer epochs. If architecture A beats architecture B on 10% of ImageNet, it likely holds true on the full dataset. The proxy reveals relative rankings quickly, then you fully train only the top candidates.
Combining Strategies
Modern NAS methods use all three together. One-Shot NAS with weight sharing (lesson 2699) is essentially extreme weight inheritance—all architectures share a supernet's weights and need zero separate training for estimation.
Key Takeaway: Performance estimation strategies make NAS practical by replacing expensive full training with cheap approximations: early stopping cuts training time, weight inheritance reduces convergence time, and proxy tasks shrink dataset/model complexity.