Course contentsShow
← Machine Learning and Deep Learning
Lesson 2700 of 3,53858. Model Compression: Pruning and DistillationFree lesson

Performance Estimation Strategies

Accelerating NAS with early stopping, weight inheritance, and proxy tasks instead of full training.

Performance Estimation Strategies in NAS

What you'll learn: How to speed up Neural Architecture Search by avoiding full training for every candidate architecture.

The Core Problem

In Neural Architecture Search, you need to compare potentially thousands of architectures. Training each one from scratch to convergence would take months or years of compute time. Performance estimation strategies let you predict which architectures are promising without wasting resources on full training.

Three Key Acceleration Techniques

Early Stopping

Train each candidate for just a few epochs instead of full convergence. Architectures that perform poorly early are likely to remain poor performers—so stop them and move on. Think of it like a cooking competition: a chef struggling with basic knife skills in the first 5 minutes probably won't win the full hour-long challenge.

Weight Inheritance

Instead of random initialization, inherit weights from a previously evaluated similar architecture. If you've just tested a 5-layer network and now want to test a 6-layer variant, copy the first 5 layers' weights. This gives the new architecture a "warm start" and reaches meaningful performance faster.

Proxy Tasks

Train on simplified versions of your target task: fewer training samples, lower resolution images, or fewer epochs. If architecture A beats architecture B on 10% of ImageNet, it likely holds true on the full dataset. The proxy reveals relative rankings quickly, then you fully train only the top candidates.

Combining Strategies

Modern NAS methods use all three together. One-Shot NAS with weight sharing (lesson 2699) is essentially extreme weight inheritance—all architectures share a supernet's weights and need zero separate training for estimation.

Key Takeaway: Performance estimation strategies make NAS practical by replacing expensive full training with cheap approximations: early stopping cuts training time, weight inheritance reduces convergence time, and proxy tasks shrink dataset/model complexity.