Fully Connected Layers: Dense Connections
What You'll Learn
You'll understand how neurons in fully connected (dense) layers wire up to every neuron in the previous layer, and why this creates a massive number of parameters.
The Core Idea
In a fully connected layer (also called a dense layer), every single neuron connects to every single neuron in the previous layer. No connections are skipped—it's an "all-to-all" wiring scheme.
Think of it like a social network where everyone is friends with everyone else. If you have 100 people in one room and 50 in another, and each person in the second room must shake hands with every person in the first room, you'll have 100 × 50 = 5,000 handshakes. Those handshakes are like the weights connecting neurons.
Parameter Count Implications
This dense connectivity comes at a cost: parameters explode quickly.
If layer i has n neurons and layer i+1 has m neurons, a fully connected layer between them requires:
- Weights:
n × m(one weight per connection) - Biases:
m(one bias per neuron in the new layer) - Total parameters:
n × m + m
Example
Suppose you have:
- Input layer: 784 neurons (like a 28×28 pixel image)
- First hidden layer: 128 neurons
The fully connected layer needs 784 × 128 + 128 = 100,480 parameters! Add more layers with hundreds or thousands of neurons, and you can easily reach millions of parameters.
This dense structure gives the network tremendous flexibility to learn complex patterns, but it also demands more memory, computation, and data to train effectively.
Key Takeaway: Fully connected layers connect every neuron to all neurons in the previous layer, creating n × m weights plus biases—powerful but parameter-heavy architectures.