Implementing LoRA in PyTorch
What you'll learn: How to code a simple LoRA layer in PyTorch that wraps a linear layer with low-rank adaptation matrices.
The Core Idea
Now that you understand the theory behind LoRA (freezing base weights and adding low-rank updates), let's build it. A LoRA layer wraps around an existing nn.Linear layer, freezes its original weight matrix W, and adds two small trainable matrices A and B whose product forms the low-rank update. During the forward pass, you compute output = x @ W + x @ (B @ A).
Basic LoRA Layer Implementation
Here's a minimal LoRA wrapper:
import torch
import torch.nn as nn
class LoRALayer(nn.Module):
def __init__(self, base_layer, rank=4, alpha=16):
super().__init__()
self.base_layer = base_layer
self.rank = rank
self.alpha = alpha
# Freeze the base layer
for param in self.base_layer.parameters():
param.requires_grad = False
# Low-rank matrices: A (in_features → rank), B (rank → out_features)
in_features = base_layer.in_features
out_features = base_layer.out_features
self.lora_A = nn.Parameter(torch.randn(in_features, rank))
self.lora_B = nn.Parameter(torch.zeros(rank, out_features))
self.scaling = alpha / rank
def forward(self, x):
# Base output (frozen) + low-rank adaptation
base_output = self.base_layer(x)
lora_output = (x @ self.lora_A @ self.lora_B) * self.scaling
return base_output + lora_output
You initialize A randomly and B to zeros (so the adapter starts with zero effect). The scaling factor controls the contribution magnitude.
Key Takeaway: A LoRA layer in PyTorch freezes the original linear weights and adds two small trainable matrices (lora_A and lora_B) whose product forms a low-rank update, drastically reducing trainable parameters while preserving adaptation capacity.