Course contentsShow
← Machine Learning and Deep Learning
Lesson 1737 of 3,53837. LLM Fine-Tuning: Full and Parameter-EfficientFree lesson

Adapter Layers: Architecture and Motivation

Learn how adapter modules insert small trainable bottleneck layers into frozen pretrained models for efficient fine-tuning.

Adapter Layers: Architecture and Motivation

What you'll learn: How adapter modules enable efficient fine-tuning by inserting small trainable bottleneck layers into frozen pretrained models.

The Core Problem

After covering LoRA and QLoRA, you've seen parameter-efficient methods that modify existing weight matrices. Adapter layers take a different architectural approach: they insert entirely new small modules between frozen pretrained layers rather than modifying weights in-place.

How Adapters Work

Think of adapters like adding expansion joints to a rigid pipeline. The original model remains completely frozen, but you insert tiny trainable "detours" that learn task-specific transformations:

  1. Down-projection: Compress the layer's output from dimension d to bottleneck dimension r (where r << d)
  2. Non-linearity: Apply an activation function (typically ReLU or GELU)
  3. Up-projection: Expand back from r to original dimension d
  4. Residual connection: Add the adapter output to the original frozen layer output

The bottleneck dimension r might be 64 or 128, while d could be 4096—training only ~1% of parameters.

Why This Architecture Matters

Unlike LoRA's rank decomposition approach, adapters create independent learned pathways through the network. The residual connection ensures the original model's knowledge is preserved (if adapter weights initialize to zero, the model starts unchanged). Each adapter can specialize to your task without touching pretrained weights.

The modular design means you can train multiple adapters for different tasks and swap them at inference time—similar to LoRA's multi-adapter capability but with different architectural constraints.

Key Takeaway: Adapter layers achieve parameter efficiency by inserting small trainable bottleneck modules between frozen layers with residual connections, offering a modular alternative to weight-modification methods like LoRA.