From Theory to Code: Implementing Backpropagation
What you'll learn: How to translate the mathematical backpropagation derivations you've studied into working Python code with correct array shapes and indexing.
Bridging Theory and Practice
You've learned the chain rule, gradient derivations, and matrix formulations—now it's time to write actual code. The biggest challenge isn't the math itself, but ensuring your arrays have the right dimensions and getting the indexing right. A single transposed matrix or swapped dimension can break everything silently.
Critical Implementation Considerations
Shape Tracking
Each layer's weight matrix W has shape (output_dim, input_dim). When computing gradients, you must ensure dW matches this exactly. If your forward pass computes Z = W @ X, then the gradient dW = dZ @ X.T (where X.T is the transpose).
Batch Dimensions
With batched inputs of shape (batch_size, features), your gradients accumulate across the batch. Remember to average: dW = (1/batch_size) * dX.T @ dZ.
Activation Function Derivatives
Your sigmoid_derivative(z) or relu_derivative(z) must operate element-wise and preserve input shape. These multiply into your gradient flow: dZ_prev = dA * activation_derivative(Z).
Backwards Flow
Start from the loss gradient at the output, then propagate backwards layer-by-layer. Each layer receives dA (gradient w.r.t. its activations) and must compute:
dW: gradient for its weightsdb: gradient for its biasesdA_prev: gradient to pass to the previous layer
Debugging Strategy
Always print shapes at each step initially. Use gradient checking (comparing numerical vs analytical gradients) to verify correctness before trusting your implementation.
Key Takeaway: Successful backpropagation implementation requires meticulous attention to matrix dimensions, proper transposition, and batch averaging—the math is one thing, but correct indexing is where theory meets reality.