Course contentsShow
← Machine Learning and Deep Learning
Lesson 2695 of 3,53858. Model Compression: Pruning and DistillationFree lesson

NAS Search Strategies: Grid and Random Search

Exploring exhaustive grid search and random sampling as baseline NAS strategies and their limitations.

NAS Search Strategies: Grid and Random Search

What you'll learn: How grid search and random sampling serve as baseline NAS approaches, and why they quickly become impractical for neural architecture search.

The Simplest Search Strategies

Now that you understand the NAS search space (the set of all possible architectures you could build), you need a search strategy to actually explore it and find good architectures. The two most straightforward baseline approaches are grid search and random search.

Grid Search: Exhaustive but Expensive

Grid search systematically evaluates every possible combination of architectural choices in your search space. If you have 3 options for number of layers, 4 options for layer width, and 2 activation functions, grid search trains and evaluates all 3 × 4 × 2 = 24 architectures.

The problem: NAS search spaces are typically enormous. With even modest choices (5 layer types, 10 possible widths, 20 depths), you'd have thousands or millions of architectures. Training each one fully is computationally prohibitive—imagine training 10,000 neural networks from scratch!

Random Search: Surprisingly Competitive

Random search samples architectures uniformly at random from the search space, trains them, and picks the best performer. Counterintuitively, research shows that random search often performs nearly as well as sophisticated methods when you have a limited budget.

Think of it like finding a good restaurant: systematically visiting every restaurant (grid search) takes forever, but randomly trying a dozen spots often gets you something satisfactory.

Why Both Are Baselines, Not Solutions

Both strategies treat architectures as black boxes and ignore valuable information like gradients or performance trends. They're useful for establishing performance benchmarks but rarely used in practice due to their computational cost and inability to scale to complex search spaces.

Key Takeaway: Grid and random search provide simple baseline NAS strategies, but their inability to leverage architectural insights and their exponential cost with search space size motivate the need for smarter search algorithms.