Course contentsShow
← Machine Learning and Deep Learning
Lesson 1094 of 3,53824. The Transformer ArchitectureFree lesson

The Encoder Stack

How multiple encoder layers transform input through self-attention and feed-forward networks.

The Encoder Stack

What you'll learn: How the Transformer encoder builds deep representations by stacking multiple identical layers that process input through self-attention and feed-forward networks.

The Core Architecture

Think of the encoder stack like a multi-story building where each floor refines and enriches information from the floor below. In Transformers, the encoder stack consists of N identical layers (typically 6-12) stacked on top of each other.

Each encoder layer contains two main sub-components:

1. Multi-Head Self-Attention

The input sequence attends to itself, capturing relationships between all positions. Each token learns contextual information from every other token in the sequence.

2. Position-wise Feed-Forward Network

After attention, each position independently passes through the same feed-forward network (two linear transformations with a non-linearity between them). This processes the attended information and adds representational capacity.

How Information Flows

The input embeddings (with positional encodings added) enter the first encoder layer. Each layer transforms its input and passes the output to the next layer:

  • Layer 1 learns basic patterns and relationships
  • Layer 2 builds on those patterns, finding more complex dependencies
  • Deeper layers capture increasingly abstract representations

Crucially, each sub-component uses residual connections (adding the input back to the output) and layer normalization to stabilize training and enable deep stacking.

Why Stack Multiple Layers?

Just like deep CNNs learn hierarchical features (edges → textures → objects), stacked encoders learn hierarchical sequence patterns. Early layers might capture local syntax, while deeper layers understand semantic relationships and global context.

Key Takeaway: The encoder stack transforms input sequences through multiple layers of self-attention and feed-forward processing, with each layer building progressively richer representations that capture both local and global dependencies.