Course contentsShow
← Machine Learning and Deep Learning
Lesson 71 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Covariance and Correlation

Measure linear relationships between random variables using covariance and correlation coefficients.

Covariance and Correlation

What you'll learn: How to measure whether two random variables tend to move together or in opposite directions, and by how much.

The Core Idea

Imagine tracking study hours and exam scores for students. Do students who study more tend to score higher? Covariance measures this tendency for two variables to change together. If one variable increases when the other increases, covariance is positive. If one decreases when the other increases, it's negative.

However, covariance values are hard to interpret because they depend on the units of measurement. If we measure study hours in minutes instead of hours, covariance changes dramatically!

Correlation solves this by normalizing covariance to always fall between -1 and +1:

  • +1: perfect positive relationship (as X increases, Y increases proportionally)
  • 0: no linear relationship
  • -1: perfect negative relationship (as X increases, Y decreases proportionally)

The Math (Simplified)

For random variables X and Y with means μₓ and μᵧ:

Covariance: Cov(X,Y) = E[(X - μₓ)(Y - μᵧ)]

Think of it as: "On average, how much do deviations from each mean multiply together?"

Correlation coefficient (ρ): ρ = Cov(X,Y) / (σₓ · σᵧ)

This divides covariance by both standard deviations, creating a scale-free measure.

Why It Matters

In machine learning, correlation helps identify which features move together, revealing potential redundancies or relationships. Before building models, understanding these linear relationships guides feature selection and helps detect multicollinearity.

Key Takeaway: Covariance measures how two variables vary together, while correlation standardizes this to a -1 to +1 scale, making linear relationships easy to interpret regardless of measurement units.