Evolutionary Algorithms for NAS
You'll learn how genetic algorithms evolve entire populations of neural architectures by mimicking natural selection through mutation, crossover, and fitness-based survival.
The Core Idea
Instead of using reinforcement learning (as in the previous lesson), evolutionary algorithms treat neural architecture search as a biological evolution problem. You maintain a population of candidate architectures, evaluate their "fitness" (validation accuracy), and breed the best performers to create the next generation.
Think of it like dog breeding: you select the fastest dogs, let them produce offspring with mixed traits (crossover), occasionally introduce random changes (mutation), and repeat until you get champion runners.
How It Works
- Initialize population: Start with random architectures from your search space
- Evaluate fitness: Train each architecture briefly and measure validation performance
- Selection: Keep top performers (e.g., top 20%)
- Crossover: Combine two parent architectures—e.g., take conv layers from parent A, skip connections from parent B
- Mutation: Randomly modify offspring (change kernel size, add/remove layers, swap activation functions)
- Repeat: New population becomes parents for the next generation
Why This Matters
Evolutionary algorithms excel at exploring diverse solutions without getting stuck in local optima. Unlike RL-based NAS (which follows a single policy), you maintain multiple promising candidates simultaneously. The mutation operator ensures continuous exploration, while crossover exploits good building blocks from successful architectures.
Trade-off: Training hundreds of architectures across many generations is computationally expensive, though techniques like weight inheritance and early stopping help.
Key Takeaway: Evolutionary algorithms for NAS use genetic programming—populations, crossover, and mutation—to iteratively breed better-performing neural architectures, offering robust exploration at the cost of computational resources.