Original Transformer Implementation Details
You'll learn the specific architectural choices made in the landmark "Attention Is All You Need" paper that defined the Transformer model.
The Blueprint That Started It All
When Vaswani et al. introduced the Transformer in 2017, they didn't just propose a theory—they provided concrete implementation details that became the de facto standard. Understanding these original choices helps you recognize why certain numbers appear repeatedly in modern architectures.
The Core Specifications
The original Transformer used these key parameters:
- 6 layers: Both the encoder and decoder stacks contained exactly 6 identical layers each
- 8 attention heads: Each multi-head attention module split attention across 8 parallel heads
- 512 dimensions: The model dimension (d_model) was set to 512, meaning all layer inputs and outputs had this size
- 64-dimensional heads: Since 512 ÷ 8 = 64, each attention head operated in 64-dimensional space
- 2048 feedforward dimensions: The inner layer of the position-wise feedforward network expanded to 2048 dimensions before projecting back to 512
Why These Numbers?
These weren't arbitrary. The authors balanced:
- Computational efficiency: 512 dimensions with 8 heads was trainable on 2017 hardware
- Expressiveness: 6 layers provided enough depth for learning complex patterns
- Head diversity: 8 heads allowed different attention patterns without excessive computation
Think of it like a recipe that worked perfectly on first try—these proportions became the template others followed and scaled up or down based on their needs.
Key Takeaway: The original Transformer used 6 encoder layers, 6 decoder layers, 8 attention heads, and 512 model dimensions—architectural choices that balanced performance and computational feasibility, becoming the foundation for modern language models.