NAS Search Strategies: Grid and Random Search
What you'll learn: How grid search and random sampling serve as baseline NAS approaches, and why they quickly become impractical for neural architecture search.
The Simplest Search Strategies
Now that you understand the NAS search space (the set of all possible architectures you could build), you need a search strategy to actually explore it and find good architectures. The two most straightforward baseline approaches are grid search and random search.
Grid Search: Exhaustive but Expensive
Grid search systematically evaluates every possible combination of architectural choices in your search space. If you have 3 options for number of layers, 4 options for layer width, and 2 activation functions, grid search trains and evaluates all 3 × 4 × 2 = 24 architectures.
The problem: NAS search spaces are typically enormous. With even modest choices (5 layer types, 10 possible widths, 20 depths), you'd have thousands or millions of architectures. Training each one fully is computationally prohibitive—imagine training 10,000 neural networks from scratch!
Random Search: Surprisingly Competitive
Random search samples architectures uniformly at random from the search space, trains them, and picks the best performer. Counterintuitively, research shows that random search often performs nearly as well as sophisticated methods when you have a limited budget.
Think of it like finding a good restaurant: systematically visiting every restaurant (grid search) takes forever, but randomly trying a dozen spots often gets you something satisfactory.
Why Both Are Baselines, Not Solutions
Both strategies treat architectures as black boxes and ignore valuable information like gradients or performance trends. They're useful for establishing performance benchmarks but rarely used in practice due to their computational cost and inability to scale to complex search spaces.
Key Takeaway: Grid and random search provide simple baseline NAS strategies, but their inability to leverage architectural insights and their exponential cost with search space size motivate the need for smarter search algorithms.