Implementing Multi-Head Attention in PyTorch
What you'll learn: How to build a complete multi-head attention module using efficient batched operations and proper tensor reshaping in PyTorch.
The Core Implementation Challenge
You've learned the theory of multi-head attention—now it's time to code it efficiently. The key challenge is managing tensor dimensions: you need to split your d_model dimension into num_heads parallel attention operations, compute each head independently, then concatenate results back together. PyTorch's tensor reshaping and batched matrix multiplication make this elegant.
Building the Module
Here's the essential structure:
import torch
import torch.nn as nn
import math
class MultiHeadAttention(nn.Module):
def __init__(self, d_model, num_heads):
super().__init__()
assert d_model % num_heads == 0
self.d_model = d_model
self.num_heads = num_heads
self.d_k = d_model // num_heads
# Linear projections
self.W_q = nn.Linear(d_model, d_model)
self.W_k = nn.Linear(d_model, d_model)
self.W_v = nn.Linear(d_model, d_model)
self.W_o = nn.Linear(d_model, d_model)
def forward(self, Q, K, V, mask=None):
batch_size = Q.size(0)
# Project and reshape: (batch, seq_len, d_model) -> (batch, num_heads, seq_len, d_k)
Q = self.W_q(Q).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
K = self.W_k(K).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
V = self.W_v(V).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
# Scaled dot-product attention
scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_k)
if mask is not None:
scores = scores.masked_fill(mask == 0, -1e9)
attn_weights = torch.softmax(scores, dim=-1)
attn_output = torch.matmul(attn_weights, V)
# Concatenate heads: (batch, num_heads, seq_len, d_k) -> (batch, seq_len, d_model)
attn_output = attn_output.transpose(1, 2).contiguous().view(batch_size, -1, self.d_model)
return self.W_o(attn_output)
The .view() and .transpose() operations efficiently split dimensions for parallel head computation, while .contiguous() ensures proper memory layout before concatenation.
Key Takeaway: Multi-head attention in PyTorch leverages tensor reshaping to process all heads in parallel—project, split into heads, compute attention, then concatenate and project again.