Building Blocks: Conv-BN-ReLU Patterns
What You'll Learn
You'll learn the standard recipe for stacking convolutional layers with batch normalization and activation functions to create reliable CNN building blocks.
The Standard Pattern
In modern CNN architectures, we rarely use a convolutional layer by itself. Instead, we follow a proven ordering pattern that combines three operations:
Conv → BatchNorm → ReLU
This sequence has become the de facto standard because each component serves a complementary purpose:
- Convolution extracts features through learnable filters
- Batch Normalization stabilizes and accelerates training by normalizing activations
- ReLU (or other activation) introduces non-linearity for learning complex patterns
Why This Order?
Think of it like an assembly line:
- The convolution produces raw feature values (which can vary wildly in scale)
- Batch normalization normalizes these values to a consistent range before activation
- The activation function then applies the non-linearity to these normalized values
Applying batch normalization before the activation function has become standard practice because it normalizes the pre-activation values, allowing the activation to work on a consistent scale.
Common Variations
While Conv-BN-ReLU is most common, you might see:
- Conv-ReLU (older architectures, no batch norm)
- Conv-BN-ReLU-Dropout (adding spatial dropout for regularization)
- Conv-BN-LeakyReLU (using alternative activations)
The key insight is that these three layers work together as a single conceptual unit that you'll repeat multiple times throughout your network architecture.
Key Takeaway: The Conv-BN-ReLU pattern is the fundamental building block of modern CNNs, combining feature extraction, normalization, and non-linearity in a proven, repeatable sequence.