Forward Pass: Computing Activations Layer by Layer
What you'll learn: How neural networks systematically compute outputs by processing data through each layer sequentially, storing intermediate results needed for training.
The Core Idea
The forward pass is the sequential computation that transforms your input data into predictions. Think of it like an assembly line: raw materials (input) enter at one end, each station (layer) performs its specific transformation, and the final product (prediction) emerges at the other end. Crucially, we must keep track of what happens at each station because we'll need this information later to improve the process.
Step-by-Step Computation
When performing a forward pass, you:
- Start with input data
X(your features) - Compute first hidden layer: Apply weights, add bias, apply activation function → store result as
h₁ - Feed
h₁to next layer: Repeat the transformation → store result ash₂ - Continue through all layers until you reach the output
- Store every intermediate activation (
h₁,h₂, ...,output)
Each layer's computation follows the pattern:
z = W @ a_prev + b # Linear transformation
a = activation(z) # Nonlinear activation
Where a_prev is the previous layer's activation (or input for the first layer).
Why Store Activations?
You might wonder why we keep all these intermediate values. The answer lies in the computational graph: during backpropagation, we'll need these activations to calculate gradients efficiently using the chain rule. Without storing them, we'd have to recompute everything—extremely wasteful!
Key Takeaway: The forward pass computes network outputs layer by layer, storing all intermediate activations as checkpoints that enable efficient gradient computation during the backward pass.