Width vs Depth Trade-offs
What you'll learn: How to think about making models wider (more neurons per layer) versus deeper (more layers) when you have a fixed parameter budget.
The Fundamental Trade-off
When designing large language models, you face a critical architectural decision: given a target parameter count (say, 1 billion parameters), should you build a wide and shallow model or a narrow and deep model?
A wide shallow model has fewer layers but many neurons (or attention heads) per layer. Think of it like a short, thick tree trunk. A narrow deep model has many layers but fewer neurons per layer—like a tall, thin tower.
Why This Matters
At the same parameter count, these architectures behave very differently:
-
Deep models excel at learning hierarchical representations. Each layer can transform information in increasingly abstract ways, making them better at complex reasoning and compositional tasks. However, they're harder to train (vanishing gradients, optimization challenges) and slower to run (sequential bottleneck).
-
Wide models offer more parallelism—computation within a layer can happen simultaneously. They're often easier to optimize and can capture more diverse patterns at each level. But they may struggle with tasks requiring many reasoning steps or deep feature hierarchies.
Practical Considerations
Modern LLMs like GPT tend toward deeper architectures (96+ layers) because language understanding benefits from hierarchical abstraction. However, research shows diminishing returns: doubling depth doesn't double performance. Recent work explores hybrid approaches—moderately deep with increased width—as a sweet spot for both expressiveness and trainability.
Key Takeaway: At fixed parameter counts, depth enables hierarchical reasoning while width increases representational diversity; modern LLMs typically favor depth for language tasks, but the optimal balance depends on your specific application and computational constraints.