The NAS Search Space
What you'll learn: You'll understand what architectural choices Neural Architecture Search can optimize when designing networks automatically.
What Is a Search Space?
In Neural Architecture Search, the search space defines the universe of possible architectures that can be explored. Think of it like a massive Lego set—the search space determines which pieces you have available and how they can connect together.
When you performed manual architecture design in earlier lessons, you made countless choices: how many layers? What operations? Which connections? NAS automates this, but it needs boundaries. The search space is those boundaries.
The Four Pillars of NAS Search Space
1. Layers
How many layers can the network have? What's the minimum and maximum depth? Can different branches have different depths?
2. Operations
What can each layer do? Options might include:
- Convolutions (3×3, 5×5, separable)
- Pooling (max, average)
- Skip connections (identity)
- Fully connected layers
3. Connections
How can layers connect? Sequential only? Skip connections allowed? Dense connections? Think of this as the network's "wiring diagram."
4. Hyperparameters
For each operation, what settings are tunable? Number of filters, kernel sizes, activation functions, expansion ratios—all the knobs you've manually tuned before.
Why It Matters
A narrow search space (e.g., only varying filter counts) runs fast but might miss innovative designs. A wide search space (e.g., every possible connection) can discover breakthroughs but takes enormous compute. This is the fundamental trade-off you face before running NAS.
Key Takeaway: The NAS search space defines which layers, operations, connections, and hyperparameters can be optimized—it's the design playground where automated architecture discovery happens.