Course contentsShow
← Machine Learning and Deep Learning
Lesson 611 of 3,53815. Neural Network FoundationsFree lesson

Numerical Stability in Forward Pass

Handling numerical issues like overflow and underflow during forward propagation, especially in deep networks.

Numerical Stability in Forward Pass

What you'll learn: How to prevent catastrophic numerical errors that can derail deep neural networks during forward propagation.

The Problem: When Numbers Break Down

When you perform forward propagation through deep networks, you're computing matrix multiplications and applying activation functions layer after layer. These repeated computations can push numbers toward extreme values that computers can't properly represent.

Overflow occurs when numbers become so large they exceed what your computer can store (imagine multiplying 10^300 × 10^300). Underflow happens when numbers get so tiny they round down to zero (like 10^-300 × 10^-300). Both situations break your network's calculations.

Why Deep Networks Are Vulnerable

Think of forward propagation like a game of telephone played with numbers. Each layer slightly distorts the values. In shallow networks, this isn't a problem. But in networks with dozens or hundreds of layers, small distortions compound dramatically.

Common culprits:

  • Exponential functions (like in softmax or sigmoid) can explode to infinity
  • Very large or small weight values that multiply through many layers
  • Unbounded activations that grow without limit

Practical Safeguards

Normalization techniques keep intermediate activations in reasonable ranges between layers. Careful weight initialization prevents values from growing or shrinking exponentially from the start. Gradient clipping (though primarily for backpropagation) also helps maintain stability.

For operations like softmax that compute exponentials, the log-sum-exp trick subtracts the maximum value before exponentiating—this prevents overflow while keeping results mathematically identical.

Key Takeaway: Numerical stability requires proactive design choices—proper initialization, normalization, and numerical tricks—to prevent overflow and underflow from corrupting forward propagation in deep networks.