Model Surgery: Freezing Layers and Transfer Learning Setup
What You'll Learn
You'll learn how to selectively freeze specific layers in a neural network by controlling the requires_grad attribute, enabling efficient transfer learning and fine-tuning workflows.
The Core Idea
When using a pre-trained model, you often want to keep some layers unchanged while training others. Think of it like renovating a house: you might keep the foundation and walls intact while updating only the interior decorations. In neural networks, freezing means preventing certain layers from updating their weights during training.
You control this behavior through the requires_grad attribute you learned earlier. When you set requires_grad=False for a parameter, PyTorch's autograd system stops computing gradients for it, and the optimizer won't update those weights.
Why Freeze Layers?
In transfer learning, early layers of a pre-trained model have already learned useful low-level features (like edges in images). You typically want to preserve this knowledge while training only the later layers to adapt to your specific task. This is faster, requires less data, and often produces better results than training from scratch.
How to Freeze Layers
# Freeze all parameters in a model
for param in model.parameters():
param.requires_grad = False
# Freeze only specific layers (e.g., first two layers)
for param in model.layer1.parameters():
param.requires_grad = False
for param in model.layer2.parameters():
param.requires_grad = False
# Keep the final layer trainable
for param in model.final_layer.parameters():
param.requires_grad = True
When you call .backward(), only parameters with requires_grad=True will receive gradients, and only those will be updated by the optimizer.
Key Takeaway: Freezing layers by setting requires_grad=False lets you selectively control which parts of your model update during training, making transfer learning efficient and effective.