Course contentsShow
← Machine Learning and Deep Learning
Lesson 1738 of 3,53837. LLM Fine-Tuning: Full and Parameter-EfficientFree lesson

Implementing Adapters in Transformer Blocks

Understand where to place adapter layers in attention and feedforward blocks and how to initialize them properly.

Implementing Adapters in Transformer Blocks

What you'll learn: Where to place adapter layers within transformer blocks and how to initialize them for stable, effective fine-tuning.

Core Concept

Adapter layers are small bottleneck modules inserted into existing transformer blocks. But where exactly do they go, and how do you set them up so they don't disrupt the pre-trained model?

Placement locations:

  • After the multi-head attention sublayer (before residual addition)
  • After the feedforward network sublayer (before residual addition)

Each adapter typically appears twice per transformer block—once in the attention path, once in the feedforward path. They sit within the residual stream, so their output gets added back to the skip connection.

Initialization Strategy

The critical trick: initialize adapters near-zero. If you start with random weights, the first forward pass produces garbage that corrupts the pre-trained representations. Instead:

  • Initialize the down-projection layer normally (e.g., Xavier/Kaiming)
  • Initialize the up-projection layer to zeros or very small values

This ensures the adapter initially acts as an identity function (adds nearly nothing), preserving the original model behavior. During training, the adapter gradually learns useful task-specific transformations.

Think of it like adding a detour route to a highway: at first, the detour is closed (zero contribution), then it slowly opens as traffic patterns (gradients) show it's useful.

Typical Flow

Input → LayerNorm → Attention → Adapter₁ → Add & Norm → FFN → Adapter₂ → Add & Norm → Output

Each adapter processes the sublayer's output before it merges with the residual connection.

Key Takeaway: Adapter layers are inserted after attention and feedforward sublayers within the residual path, initialized near-zero to preserve pre-trained behavior before training begins.