Course contentsShow
← Machine Learning and Deep Learning
Lesson 1076 of 3,53824. The Transformer ArchitectureFree lesson

Implementing Multi-Head Attention in PyTorch

Building a multi-head attention module with efficient batched operations and proper tensor reshaping.

Implementing Multi-Head Attention in PyTorch

What you'll learn: How to build a complete multi-head attention module using efficient batched operations and proper tensor reshaping in PyTorch.

The Core Implementation Challenge

You've learned the theory of multi-head attention—now it's time to code it efficiently. The key challenge is managing tensor dimensions: you need to split your d_model dimension into num_heads parallel attention operations, compute each head independently, then concatenate results back together. PyTorch's tensor reshaping and batched matrix multiplication make this elegant.

Building the Module

Here's the essential structure:

import torch
import torch.nn as nn
import math

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads):
        super().__init__()
        assert d_model % num_heads == 0
        
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads
        
        # Linear projections
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)
        
    def forward(self, Q, K, V, mask=None):
        batch_size = Q.size(0)
        
        # Project and reshape: (batch, seq_len, d_model) -> (batch, num_heads, seq_len, d_k)
        Q = self.W_q(Q).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
        K = self.W_k(K).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
        V = self.W_v(V).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
        
        # Scaled dot-product attention
        scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_k)
        if mask is not None:
            scores = scores.masked_fill(mask == 0, -1e9)
        attn_weights = torch.softmax(scores, dim=-1)
        attn_output = torch.matmul(attn_weights, V)
        
        # Concatenate heads: (batch, num_heads, seq_len, d_k) -> (batch, seq_len, d_model)
        attn_output = attn_output.transpose(1, 2).contiguous().view(batch_size, -1, self.d_model)
        
        return self.W_o(attn_output)

The .view() and .transpose() operations efficiently split dimensions for parallel head computation, while .contiguous() ensures proper memory layout before concatenation.

Key Takeaway: Multi-head attention in PyTorch leverages tensor reshaping to process all heads in parallel—project, split into heads, compute attention, then concatenate and project again.