Gradient-Based NAS and DARTS
What you'll learn: How to make neural architecture search differentiable by turning discrete architectural choices into continuous, optimizable weights.
The Problem with Discrete Search
Traditional NAS methods (reinforcement learning, evolutionary algorithms) treat architecture choices as discrete decisions: "Should I add a 3×3 convolution or a 5×5 one here?" This makes the search space combinatorial and requires evaluating many candidate architectures sequentially—expensive and slow.
The DARTS Breakthrough
DARTS (Differentiable Architecture Search) revolutionized NAS by making the search process differentiable. Instead of picking one operation, DARTS keeps all possible operations and assigns each a continuous weight (like a probability). The final output becomes a weighted sum of all operations.
Analogy: Imagine choosing a restaurant. Traditional NAS tries each one separately and picks the best. DARTS orders from all restaurants simultaneously, weights each dish by how good it might be, and gradually increases the weight of the best options until one dominates.
How It Works
During search, each edge in the computational graph represents a mixture of operations (convolutions of different sizes, pooling, skip connections). These mixing weights are learned via gradient descent alongside the network weights. After training, you discretize by keeping only the operation with the highest weight per edge.
This transforms architecture search from a discrete optimization problem into a continuous one solvable with standard backpropagation—making it orders of magnitude faster than previous methods.
Key Takeaway: Gradient-based NAS like DARTS relaxes discrete architectural choices into continuous weights, enabling efficient architecture search through standard gradient descent rather than expensive trial-and-error methods.