One-Shot NAS and Weight Sharing
What you'll learn: How to find optimal neural architectures by training one massive "supernet" containing all possibilities, then extracting the best performing subnetwork.
The Core Idea
Instead of training thousands of separate neural networks from scratch (as in traditional NAS), one-shot NAS trains a single supernet once. This supernet is designed to contain all candidate architectures as subnetworks. After training, you can extract and evaluate different architectures by sampling paths through the supernet—without retraining from scratch.
Weight Sharing: The Key Mechanism
The "magic" is weight sharing: operations that appear in multiple candidate architectures share the same trained parameters. For example, if both Architecture A and Architecture B use a 3×3 convolution at layer 2, they use the same learned weights.
Think of it like a library of pre-trained Lego blocks. Instead of building and painting each unique model individually, you paint all the blocks once, then quickly assemble different structures to see which works best.
How It Works
- Define a search space of possible operations (different kernel sizes, skip connections, pooling layers)
- Build a supernet containing all operations in parallel at each layer
- Train the supernet by randomly sampling subnetworks (paths) and updating shared weights
- Search for the best architecture by evaluating different subnetworks using the shared weights
Why It Matters
One-shot NAS dramatically reduces computational cost—from thousands of GPU-days to just one training run. Methods like DARTS leverage this by making architecture search differentiable, but the weight-sharing principle remains central: train once, search efficiently.
Key Takeaway: One-shot NAS trains a single supernet with weight sharing across all candidate architectures, enabling efficient architecture search by extracting and evaluating subnetworks without repeated training from scratch.