Identity Matrix and Matrix Inverse

You'll learn: How the identity matrix acts like "1" for matrices, and how matrix inverses let you "undo" transformations—crucial for solving equations in ML.

The Identity Matrix

The identity matrix (denoted I) is a square matrix with 1s along the diagonal and 0s everywhere else. It's the matrix equivalent of the number 1 in multiplication.

For example, a 3×3 identity matrix:

I = [1  0  0]
    [0  1  0]
    [0  0  1]

When you multiply any matrix A by I, you get A back: A · I = I · A = A

Just like multiplying a number by 1 leaves it unchanged!

Matrix Inverse

The inverse of a matrix A (written A⁻¹) is the matrix that "undoes" A. When you multiply them: A · A⁻¹ = A⁻¹ · A = I

Think of it like division: if 5 × (1/5) = 1, then A times its inverse equals the identity.

When Can We Find an Inverse?

Not all matrices have inverses! A matrix must be:

• Square (same number of rows and columns)
• Non-singular (its columns/rows must be linearly independent—no redundant information)

Why This Matters for ML

Matrix inverses help solve linear systems like Ax = b (where x is unknown). If A has an inverse: x = A⁻¹b

This shows up when calculating optimal weights in linear regression, finding feature transformations, and understanding gradient descent paths. The identity matrix appears in regularization techniques and neural network initialization.

Key Takeaway: The identity matrix is the multiplicative "neutral element" for matrices; the inverse (when it exists) lets you reverse transformations and solve for unknowns in linear systems—foundational operations in machine learning algorithms.