Course contentsShow
← Machine Learning and Deep Learning
Lesson 1744 of 3,53837. LLM Fine-Tuning: Full and Parameter-EfficientFree lesson

Layer Selection and Partial Fine-Tuning

Learn strategies for selectively unfreezing specific layers or components instead of updating the entire model.

Layer Selection and Partial Fine-Tuning

You'll learn how to strategically unfreeze only specific layers or components of an LLM instead of updating the entire model during fine-tuning.

The Core Idea

Think of a large language model as a skyscraper where lower floors handle basic language understanding and upper floors specialize in complex reasoning. Full fine-tuning renovates every floor, while parameter-efficient methods like LoRA add small extensions. Partial fine-tuning takes a middle path: you selectively unlock and update only certain floors while keeping others frozen.

This strategy addresses a key question: Which parts of the model actually need to adapt to your task?

Why Layer Selection Matters

Not all layers contribute equally to task-specific adaptation. Early layers typically encode general linguistic features (syntax, basic semantics) that remain useful across tasks. Later layers often capture higher-level abstractions that benefit more from task-specific updates.

By freezing early layers and fine-tuning only the top layers, you:

  • Reduce memory and compute compared to full fine-tuning
  • Mitigate catastrophic forgetting by preserving foundational knowledge
  • Speed up training with fewer parameters to update
  • Balance adaptation with efficiency better than frozen-model approaches

Common Strategies

Top-N layers unfreezing: Update only the final N transformer blocks (e.g., last 4 of 32 layers).

Component-specific selection: Unfreeze only attention modules or only feed-forward networks across layers.

Gradual unfreezing: Start with top layers, progressively unlock deeper layers as training continues.

Task-guided selection: Use small-scale experiments to identify which layers change most for your task, then unfreeze those.

Practical Considerations

Choose more layers for tasks distant from pretraining (domain shift) and fewer for similar tasks. Monitor validation performance to avoid overfitting when too many parameters update.

Key Takeaway: Layer selection and partial fine-tuning offer a flexible middle ground between full fine-tuning and PEFT methods, letting you target exactly which model components should adapt to your specific task.