Course contentsShow
← Machine Learning and Deep Learning
Lesson 640 of 3,53815. Neural Network FoundationsFree lesson

From Theory to Code: Implementing Backpropagation

Translating mathematical derivations into working code with proper indexing and dimension handling.

From Theory to Code: Implementing Backpropagation

What you'll learn: How to translate the mathematical backpropagation derivations you've studied into working Python code with correct array shapes and indexing.

Bridging Theory and Practice

You've learned the chain rule, gradient derivations, and matrix formulations—now it's time to write actual code. The biggest challenge isn't the math itself, but ensuring your arrays have the right dimensions and getting the indexing right. A single transposed matrix or swapped dimension can break everything silently.

Critical Implementation Considerations

Shape Tracking

Each layer's weight matrix W has shape (output_dim, input_dim). When computing gradients, you must ensure dW matches this exactly. If your forward pass computes Z = W @ X, then the gradient dW = dZ @ X.T (where X.T is the transpose).

Batch Dimensions

With batched inputs of shape (batch_size, features), your gradients accumulate across the batch. Remember to average: dW = (1/batch_size) * dX.T @ dZ.

Activation Function Derivatives

Your sigmoid_derivative(z) or relu_derivative(z) must operate element-wise and preserve input shape. These multiply into your gradient flow: dZ_prev = dA * activation_derivative(Z).

Backwards Flow

Start from the loss gradient at the output, then propagate backwards layer-by-layer. Each layer receives dA (gradient w.r.t. its activations) and must compute:

  • dW: gradient for its weights
  • db: gradient for its biases
  • dA_prev: gradient to pass to the previous layer

Debugging Strategy

Always print shapes at each step initially. Use gradient checking (comparing numerical vs analytical gradients) to verify correctness before trusting your implementation.

Key Takeaway: Successful backpropagation implementation requires meticulous attention to matrix dimensions, proper transposition, and batch averaging—the math is one thing, but correct indexing is where theory meets reality.