Course contentsShow
← Machine Learning and Deep Learning
Lesson 386 of 3,5389. Dimensionality ReductionFree lesson

Covariance Matrix Construction

Computing the covariance matrix from centered data: understanding its role in capturing feature relationships.

Covariance Matrix Construction

What You'll Learn

You'll learn how to build the covariance matrix from your centered data and why it's essential for understanding how features relate to one another in Principal Component Analysis.

The Core Idea

Once you've centered your data (subtracted the mean from each feature), the next step in PCA is constructing the covariance matrix. This matrix captures how every pair of features varies together.

Think of it like a friendship chart: instead of marking who's friends with whom, you're marking how strongly each feature moves with every other feature. If two features tend to increase together, they have positive covariance. If one goes up while the other goes down, they have negative covariance.

Computing the Covariance Matrix

If your centered data matrix is X (with n samples and p features), the covariance matrix C is calculated as:

C = (1/n) * X^T * X

This gives you a p × p symmetric matrix where:

  • The diagonal contains variances (how much each feature varies on its own)
  • The off-diagonal elements contain covariances (how pairs of features vary together)

Why This Matters

The covariance matrix is the heart of PCA. It encodes all the information about feature relationships in a compact form. PCA will analyze this matrix to find directions (principal components) along which your data varies the most. Features with high covariance are redundant—they tell similar stories—and PCA exploits this to reduce dimensions while preserving information.

Key Takeaway: The covariance matrix summarizes all pairwise feature relationships in centered data, providing the foundation for PCA to identify the most informative directions in your dataset.