Course contentsShow
← Machine Learning and Deep Learning
Lesson 2697 of 3,53858. Model Compression: Pruning and DistillationFree lesson

Evolutionary Algorithms for NAS

Applying genetic algorithms with mutation and crossover to evolve populations of neural architectures.

Evolutionary Algorithms for NAS

You'll learn how genetic algorithms evolve entire populations of neural architectures by mimicking natural selection through mutation, crossover, and fitness-based survival.

The Core Idea

Instead of using reinforcement learning (as in the previous lesson), evolutionary algorithms treat neural architecture search as a biological evolution problem. You maintain a population of candidate architectures, evaluate their "fitness" (validation accuracy), and breed the best performers to create the next generation.

Think of it like dog breeding: you select the fastest dogs, let them produce offspring with mixed traits (crossover), occasionally introduce random changes (mutation), and repeat until you get champion runners.

How It Works

  1. Initialize population: Start with random architectures from your search space
  2. Evaluate fitness: Train each architecture briefly and measure validation performance
  3. Selection: Keep top performers (e.g., top 20%)
  4. Crossover: Combine two parent architectures—e.g., take conv layers from parent A, skip connections from parent B
  5. Mutation: Randomly modify offspring (change kernel size, add/remove layers, swap activation functions)
  6. Repeat: New population becomes parents for the next generation

Why This Matters

Evolutionary algorithms excel at exploring diverse solutions without getting stuck in local optima. Unlike RL-based NAS (which follows a single policy), you maintain multiple promising candidates simultaneously. The mutation operator ensures continuous exploration, while crossover exploits good building blocks from successful architectures.

Trade-off: Training hundreds of architectures across many generations is computationally expensive, though techniques like weight inheritance and early stopping help.

Key Takeaway: Evolutionary algorithms for NAS use genetic programming—populations, crossover, and mutation—to iteratively breed better-performing neural architectures, offering robust exploration at the cost of computational resources.