Course contentsShow
← Machine Learning and Deep Learning
Lesson 2694 of 3,53858. Model Compression: Pruning and DistillationFree lesson

The NAS Search Space

Defining what architectural choices can be optimized: layers, connections, operations, and hyperparameters.

The NAS Search Space

What you'll learn: You'll understand what architectural choices Neural Architecture Search can optimize when designing networks automatically.

What Is a Search Space?

In Neural Architecture Search, the search space defines the universe of possible architectures that can be explored. Think of it like a massive Lego set—the search space determines which pieces you have available and how they can connect together.

When you performed manual architecture design in earlier lessons, you made countless choices: how many layers? What operations? Which connections? NAS automates this, but it needs boundaries. The search space is those boundaries.

The Four Pillars of NAS Search Space

1. Layers

How many layers can the network have? What's the minimum and maximum depth? Can different branches have different depths?

2. Operations

What can each layer do? Options might include:

  • Convolutions (3×3, 5×5, separable)
  • Pooling (max, average)
  • Skip connections (identity)
  • Fully connected layers

3. Connections

How can layers connect? Sequential only? Skip connections allowed? Dense connections? Think of this as the network's "wiring diagram."

4. Hyperparameters

For each operation, what settings are tunable? Number of filters, kernel sizes, activation functions, expansion ratios—all the knobs you've manually tuned before.

Why It Matters

A narrow search space (e.g., only varying filter counts) runs fast but might miss innovative designs. A wide search space (e.g., every possible connection) can discover breakthroughs but takes enormous compute. This is the fundamental trade-off you face before running NAS.

Key Takeaway: The NAS search space defines which layers, operations, connections, and hyperparameters can be optimized—it's the design playground where automated architecture discovery happens.