Matrix Multiplication and Linear Algebra
What you'll learn: How to perform matrix multiplication in PyTorch using multiple methods and access linear algebra operations that power neural networks.
Why Matrix Multiplication Matters
In deep learning, nearly every operation is a matrix multiplication at its core. When data flows through a neural network layer, it's being matrix-multiplied by weights. Understanding how to perform these operations efficiently in PyTorch is fundamental.
Three Ways to Multiply Matrices
PyTorch gives you three equivalent ways to multiply matrices:
torch.mm()– The explicit matrix multiplication function (works only for 2D tensors)torch.matmul()– A more flexible function that handles batches and broadcasting@operator – Python's matrix multiplication operator (cleanest syntax)
Think of torch.mm() as the strict mathematician who only works with 2D matrices, while torch.matmul() and @ are more flexible, handling various tensor dimensions intelligently.
import torch
A = torch.tensor([[1, 2], [3, 4]])
B = torch.tensor([[5, 6], [7, 8]])
# All three produce the same result for 2D tensors:
result1 = torch.mm(A, B)
result2 = torch.matmul(A, B)
result3 = A @ B # Most Pythonic!
Linear Algebra Toolkit
For advanced operations like decompositions (breaking matrices into simpler components), PyTorch provides torch.linalg. This module contains functions for eigenvalue decomposition, matrix inversion, solving linear systems, and more – essential tools for understanding what's happening inside optimization algorithms.
# Compute the inverse of a matrix
A_inv = torch.linalg.inv(A)
Key Takeaway: Use @ for clean matrix multiplication, torch.matmul() for batch operations, and torch.linalg for decompositions and advanced linear algebra operations that underpin deep learning mathematics.