Batch Normalization: Forward Pass Mathematics
What you'll learn: How batch normalization transforms activations mathematically during the forward pass through a neural network.
The Core Idea
Batch normalization normalizes layer inputs across a mini-batch, then applies learned scale and shift parameters. The forward pass involves four mathematical steps: computing batch statistics, normalizing, scaling, and shifting.
The Forward Pass Equations
Given a mini-batch of activations x = {x₁, x₂, ..., xₘ} for a particular feature:
Step 1: Compute Batch Mean
μ_B = (1/m) Σ xᵢ
Calculate the average activation across the batch.
Step 2: Compute Batch Variance
σ²_B = (1/m) Σ (xᵢ - μ_B)²
Measure how spread out the activations are.
Step 3: Normalize
x̂ᵢ = (xᵢ - μ_B) / √(σ²_B + ε)
Transform each activation to have zero mean and unit variance. The small constant ε (typically 10⁻⁵) prevents division by zero.
Step 4: Scale and Shift
yᵢ = γ x̂ᵢ + β
Apply learnable parameters γ (scale) and β (shift) to restore representational power. Without these, the network couldn't learn the optimal distribution—sometimes the best distribution isn't zero-mean and unit-variance!
Why These Steps Matter
The normalization step (Step 3) stabilizes training, while the scale/shift parameters (Step 4) give the network flexibility. If needed, the network can even learn to undo the normalization by setting γ = √(σ²_B) and β = μ_B.
Key Takeaway: Batch normalization's forward pass normalizes using batch statistics (μ_B, σ²_B), then applies learnable transformations (γ, β) to maintain the network's expressive power while stabilizing gradients.