Depth vs Width: Architectural Trade-offs
You'll learn: How the choice between deeper versus wider neural networks affects what your model learns and how efficiently it uses parameters.
The Core Trade-off
When designing a neural network, you face a fundamental choice: should you make it deeper (more layers) or wider (more neurons per layer)?
Depth means stacking more hidden layers. Deeper networks excel at learning hierarchical features — think of it like building understanding in stages. The first layer might detect edges, the second combines edges into shapes, the third recognizes object parts, and so on. Each layer refines and abstracts the previous layer's output, creating increasingly sophisticated representations.
Width means adding more neurons within each layer. Wider networks increase the representational capacity per layer — they can capture more patterns at the same level of abstraction. A wider network can model more variations simultaneously but doesn't necessarily create the hierarchical structure that depth provides.
The Practical Difference
Imagine learning to recognize faces:
- A deep network might learn: pixels → edges → facial features → face parts → identity
- A wide network might learn: many different pixel-to-feature mappings in parallel, but at similar abstraction levels
Research shows that depth is often more parameter-efficient for complex, hierarchical problems (like vision or language). A 10-layer network with 100 neurons each can learn patterns that might require exponentially more neurons in a shallow, wide network.
However, wider networks can be easier to train (fewer vanishing gradient issues) and parallelize better across hardware.
Key Takeaway: Deep networks capture hierarchical, compositional features efficiently, while wide networks increase learning capacity within each abstraction level — modern architectures often balance both dimensions strategically.