Course contentsShow
← Machine Learning and Deep Learning
Lesson 1721 of 3,53837. LLM Fine-Tuning: Full and Parameter-EfficientFree lesson

Implementing LoRA in PyTorch

Coding a basic LoRA layer that wraps a linear layer with low-rank adaptation matrices.

Implementing LoRA in PyTorch

What you'll learn: How to code a simple LoRA layer in PyTorch that wraps a linear layer with low-rank adaptation matrices.

The Core Idea

Now that you understand the theory behind LoRA (freezing base weights and adding low-rank updates), let's build it. A LoRA layer wraps around an existing nn.Linear layer, freezes its original weight matrix W, and adds two small trainable matrices A and B whose product forms the low-rank update. During the forward pass, you compute output = x @ W + x @ (B @ A).

Basic LoRA Layer Implementation

Here's a minimal LoRA wrapper:

import torch
import torch.nn as nn

class LoRALayer(nn.Module):
    def __init__(self, base_layer, rank=4, alpha=16):
        super().__init__()
        self.base_layer = base_layer
        self.rank = rank
        self.alpha = alpha
        
        # Freeze the base layer
        for param in self.base_layer.parameters():
            param.requires_grad = False
        
        # Low-rank matrices: A (in_features → rank), B (rank → out_features)
        in_features = base_layer.in_features
        out_features = base_layer.out_features
        
        self.lora_A = nn.Parameter(torch.randn(in_features, rank))
        self.lora_B = nn.Parameter(torch.zeros(rank, out_features))
        self.scaling = alpha / rank
    
    def forward(self, x):
        # Base output (frozen) + low-rank adaptation
        base_output = self.base_layer(x)
        lora_output = (x @ self.lora_A @ self.lora_B) * self.scaling
        return base_output + lora_output

You initialize A randomly and B to zeros (so the adapter starts with zero effect). The scaling factor controls the contribution magnitude.

Key Takeaway: A LoRA layer in PyTorch freezes the original linear weights and adds two small trainable matrices (lora_A and lora_B) whose product forms a low-rank update, drastically reducing trainable parameters while preserving adaptation capacity.