Course contentsShow
← Machine Learning and Deep Learning
Lesson 79 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Covariance and Correlation

Measuring linear relationships between variables using covariance and Pearson correlation coefficient.

Covariance and Correlation

What you'll learn: How to measure and interpret the linear relationship between two variables using covariance and correlation.

The Core Idea

Imagine you're tracking two variables—say, hours studied and test scores. Do they move together? When one increases, does the other tend to increase too? Covariance and correlation quantify this relationship.

Covariance measures how two variables change together. If both tend to be above their means at the same time (or below together), covariance is positive. If one is high when the other is low, it's negative. If they don't show a pattern, it's near zero.

The formula for covariance between variables X and Y:

Cov(X,Y) = E[(X - E[X])(Y - E[Y])]

However, covariance has a problem: its value depends on the scale of your variables. Studying hours in minutes vs. hours gives different covariances!

Pearson correlation coefficient solves this by normalizing covariance. It always falls between -1 and +1:

ρ(X,Y) = Cov(X,Y) / (σ_X × σ_Y)

Where σ represents standard deviation.

Interpretation

  • ρ = +1: Perfect positive linear relationship
  • ρ = 0: No linear relationship
  • ρ = -1: Perfect negative linear relationship
  • ρ = 0.7: Strong positive relationship
  • ρ = -0.3: Weak negative relationship

Why It Matters in ML

Understanding which features are correlated helps you identify redundant information, detect patterns, and understand how variables influence each other before building models.

Key Takeaway: Covariance measures how two variables vary together, while correlation standardizes this to a -1 to +1 scale, making it easier to interpret the strength and direction of linear relationships.