Course contentsShow
← Machine Learning and Deep Learning
Lesson 633 of 3,53815. Neural Network FoundationsFree lesson

Backpropagation for Fully Connected Layers

Complete derivation of weight and bias gradients for dense layers including all matrix dimensions.

Backpropagation for Fully Connected Layers

What You'll Learn

You'll understand the complete mathematical derivation of weight and bias gradients for dense layers, including proper matrix dimensions.

The Core Idea

Backpropagation through fully connected (dense) layers computes gradients with respect to weights W and biases b so we can update them during training. The key is tracking how changes in these parameters affect the final loss through the chain rule.

For a dense layer computing z = W·x + b followed by activation a = σ(z):

Weight Gradient

The gradient of loss L with respect to weights W is:

dL/dW = dL/da · da/dz · dz/dW

Since dz/dW = x^T (the input transposed), we get:

dL/dW = (dL/dz) · x^T

Matrix dimensions: If W is (n_out × n_in), x is (n_in × 1), and dL/dz is (n_out × 1), then dL/dW is correctly (n_out × n_in).

Bias Gradient

The bias gradient is simpler because dz/db = 1:

dL/db = dL/dz

This is just the gradient flowing backward from the next layer.

Batched Computation

For a batch of m samples, inputs become X (n_in × m) and we compute:

dL/dW = (dL/dZ) · X^T / m
dL/db = sum(dL/dZ, axis=1) / m

We average gradients across the batch.

The Analogy

Think of each weight as a knob controlling signal flow. Backprop tells you exactly how much turning each knob affects your final error, accounting for all the layers the signal passed through.

Key Takeaway: Backpropagation for fully connected layers computes dL/dW = (dL/dZ)·X^T and dL/db = dL/dZ, with careful attention to matrix dimensions ensuring gradients match parameter shapes exactly.