Course contentsShow
← Machine Learning and Deep Learning
Lesson 2699 of 3,53858. Model Compression: Pruning and DistillationFree lesson

One-Shot NAS and Weight Sharing

Training a single supernet containing all candidate architectures, then extracting optimal subnetworks.

One-Shot NAS and Weight Sharing

What you'll learn: How to find optimal neural architectures by training one massive "supernet" containing all possibilities, then extracting the best performing subnetwork.

The Core Idea

Instead of training thousands of separate neural networks from scratch (as in traditional NAS), one-shot NAS trains a single supernet once. This supernet is designed to contain all candidate architectures as subnetworks. After training, you can extract and evaluate different architectures by sampling paths through the supernet—without retraining from scratch.

Weight Sharing: The Key Mechanism

The "magic" is weight sharing: operations that appear in multiple candidate architectures share the same trained parameters. For example, if both Architecture A and Architecture B use a 3×3 convolution at layer 2, they use the same learned weights.

Think of it like a library of pre-trained Lego blocks. Instead of building and painting each unique model individually, you paint all the blocks once, then quickly assemble different structures to see which works best.

How It Works

  1. Define a search space of possible operations (different kernel sizes, skip connections, pooling layers)
  2. Build a supernet containing all operations in parallel at each layer
  3. Train the supernet by randomly sampling subnetworks (paths) and updating shared weights
  4. Search for the best architecture by evaluating different subnetworks using the shared weights

Why It Matters

One-shot NAS dramatically reduces computational cost—from thousands of GPU-days to just one training run. Methods like DARTS leverage this by making architecture search differentiable, but the weight-sharing principle remains central: train once, search efficiently.

Key Takeaway: One-shot NAS trains a single supernet with weight sharing across all candidate architectures, enabling efficient architecture search by extracting and evaluating subnetworks without repeated training from scratch.