The Hyperparameter Search Space
What you'll learn: You'll understand how to define the range and types of hyperparameters your model will explore during tuning.
What Is a Hyperparameter Search Space?
The search space is the complete set of possible values you allow each hyperparameter to take when tuning your model. Think of it like a treasure map: you define the boundaries of where to search, and the tuning algorithm explores within those boundaries to find the best combination.
Before you can tune hyperparameters (which you learned are model settings you control before training), you must define what values are even worth trying.
Three Types of Search Spaces
Continuous Hyperparameters
These can take any value within a range. Example: a learning rate between 0.001 and 0.1. You might search anywhere along that continuum (0.005, 0.0237, 0.089, etc.).
Discrete Hyperparameters
These are numeric but restricted to specific intervals or integers. Example: the number of trees in a random forest must be a whole number (50, 100, 150), not 73.5.
Categorical Hyperparameters
These are non-numeric choices from a fixed set. Example: activation functions like 'relu', 'tanh', or 'sigmoid'. There's no "in-between" option—you pick one from the menu.
Why This Matters
Defining your search space well means the difference between finding a great model quickly and wasting compute time exploring irrelevant values. Too narrow, and you might miss the optimal setting. Too wide, and you'll search forever without convergence.
Key Takeaway: The hyperparameter search space defines the boundaries (continuous ranges, discrete steps, or categorical options) within which your tuning algorithm explores to find optimal model settings.