Matrix Formulation of Forward Pass
What You'll Learn
You'll learn how to express neural network forward propagation using matrix multiplication, enabling efficient computation across multiple training examples simultaneously.
The Core Idea
When we computed forward passes for single neurons or layers before, we handled one example at a time. But in practice, we train on batches of examples simultaneously. Matrix formulation lets us process entire batches in one operation.
The key formula is:
Z = WX + b
Where:
- X is the input matrix (features × examples)
- W is the weight matrix (neurons × features)
- b is the bias vector (neurons × 1)
- Z is the output before activation (neurons × examples)
How It Works
Think of it like a factory assembly line: instead of processing one item at a time, you run multiple items through simultaneously on parallel tracks.
Example dimensions:
- If you have 100 training examples with 784 features (like 28×28 pixel images)
- And a layer with 128 neurons
- Then: X is 784×100, W is 128×784, b is 128×1
- Result: Z is 128×100 (each of 100 examples gets 128 outputs)
The bias vector b is automatically broadcast (copied) across all examples—each example gets the same bias added.
Why This Matters
Matrix operations are vastly faster on modern hardware (GPUs, optimized CPUs) than looping through examples one by one. This single formula replaces what could be thousands of individual calculations with one highly optimized operation.
After computing Z, you'd apply an activation function element-wise: A = activation(Z), giving you the layer's final output.
Key Takeaway: Matrix formulation Z = WX + b transforms forward propagation from sequential processing into efficient batch computation, making neural network training practical at scale.