Backpropagation for Fully Connected Layers
What You'll Learn
You'll understand the complete mathematical derivation of weight and bias gradients for dense layers, including proper matrix dimensions.
The Core Idea
Backpropagation through fully connected (dense) layers computes gradients with respect to weights W and biases b so we can update them during training. The key is tracking how changes in these parameters affect the final loss through the chain rule.
For a dense layer computing z = W·x + b followed by activation a = σ(z):
Weight Gradient
The gradient of loss L with respect to weights W is:
dL/dW = dL/da · da/dz · dz/dW
Since dz/dW = x^T (the input transposed), we get:
dL/dW = (dL/dz) · x^T
Matrix dimensions: If W is (n_out × n_in), x is (n_in × 1), and dL/dz is (n_out × 1), then dL/dW is correctly (n_out × n_in).
Bias Gradient
The bias gradient is simpler because dz/db = 1:
dL/db = dL/dz
This is just the gradient flowing backward from the next layer.
Batched Computation
For a batch of m samples, inputs become X (n_in × m) and we compute:
dL/dW = (dL/dZ) · X^T / m
dL/db = sum(dL/dZ, axis=1) / m
We average gradients across the batch.
The Analogy
Think of each weight as a knob controlling signal flow. Backprop tells you exactly how much turning each knob affects your final error, accounting for all the layers the signal passed through.
Key Takeaway: Backpropagation for fully connected layers computes dL/dW = (dL/dZ)·X^T and dL/db = dL/dZ, with careful attention to matrix dimensions ensuring gradients match parameter shapes exactly.