Course contentsShow
← Machine Learning and Deep Learning
Lesson 753 of 3,53818. Regularization Techniques for Deep NetworksFree lesson

Batch Normalization: Forward Pass Mathematics

Derive the forward pass equations: computing batch statistics, normalization, and learnable scale/shift parameters.

Batch Normalization: Forward Pass Mathematics

What you'll learn: How batch normalization transforms activations mathematically during the forward pass through a neural network.


The Core Idea

Batch normalization normalizes layer inputs across a mini-batch, then applies learned scale and shift parameters. The forward pass involves four mathematical steps: computing batch statistics, normalizing, scaling, and shifting.

The Forward Pass Equations

Given a mini-batch of activations x = {x₁, x₂, ..., xₘ} for a particular feature:

Step 1: Compute Batch Mean

μ_B = (1/m) Σ xᵢ

Calculate the average activation across the batch.

Step 2: Compute Batch Variance

σ²_B = (1/m) Σ (xᵢ - μ_B)²

Measure how spread out the activations are.

Step 3: Normalize

x̂ᵢ = (xᵢ - μ_B) / √(σ²_B + ε)

Transform each activation to have zero mean and unit variance. The small constant ε (typically 10⁻⁵) prevents division by zero.

Step 4: Scale and Shift

yᵢ = γ x̂ᵢ + β

Apply learnable parameters γ (scale) and β (shift) to restore representational power. Without these, the network couldn't learn the optimal distribution—sometimes the best distribution isn't zero-mean and unit-variance!

Why These Steps Matter

The normalization step (Step 3) stabilizes training, while the scale/shift parameters (Step 4) give the network flexibility. If needed, the network can even learn to undo the normalization by setting γ = √(σ²_B) and β = μ_B.

Key Takeaway: Batch normalization's forward pass normalizes using batch statistics (μ_B, σ²_B), then applies learnable transformations (γ, β) to maintain the network's expressive power while stabilizing gradients.