Network Architecture Terminology: Depth and Width
What You'll Learn
You'll learn how to describe neural network architectures using two fundamental dimensions: depth (how many layers) and width (how many neurons per layer).
The Core Idea
When we describe a neural network's architecture, we use precise terminology to communicate its structure:
Depth refers to the number of layers in your network. Count all layers that process information: input layer, hidden layers, and output layer. A network with 1 input layer, 3 hidden layers, and 1 output layer has a depth of 5. The term "deep learning" literally comes from using networks with many layers—high depth.
Width refers to how many neurons exist in a single layer. Different layers can have different widths. For example, your first hidden layer might have 128 neurons, while your second has 64.
Architecture Notation
We commonly describe networks using notation like:
- 784-128-64-10: Input layer (784 neurons), two hidden layers (128 and 64 neurons), output layer (10 neurons)
- This network has depth = 4 and varying width per layer
Think of depth like floors in a building—more floors mean taller (deeper). Width is like the size of each floor—more rooms (neurons) mean wider processing capacity at that level.
Why It Matters
Depth and width control your network's capacity to learn complex patterns. Deeper networks can learn hierarchical features (edges → shapes → objects), while wider layers can capture more diverse features at each level. The universal approximation theorem tells us even shallow networks work, but deeper architectures often learn more efficiently.
Key Takeaway: Network depth counts the total layers, width measures neurons per layer, and together they define your architecture's learning capacity and computational cost.