Covariance and Correlation
What you'll learn: How to measure whether two random variables tend to move together or in opposite directions, and by how much.
The Core Idea
Imagine tracking study hours and exam scores for students. Do students who study more tend to score higher? Covariance measures this tendency for two variables to change together. If one variable increases when the other increases, covariance is positive. If one decreases when the other increases, it's negative.
However, covariance values are hard to interpret because they depend on the units of measurement. If we measure study hours in minutes instead of hours, covariance changes dramatically!
Correlation solves this by normalizing covariance to always fall between -1 and +1:
- +1: perfect positive relationship (as X increases, Y increases proportionally)
- 0: no linear relationship
- -1: perfect negative relationship (as X increases, Y decreases proportionally)
The Math (Simplified)
For random variables X and Y with means μₓ and μᵧ:
Covariance: Cov(X,Y) = E[(X - μₓ)(Y - μᵧ)]
Think of it as: "On average, how much do deviations from each mean multiply together?"
Correlation coefficient (ρ): ρ = Cov(X,Y) / (σₓ · σᵧ)
This divides covariance by both standard deviations, creating a scale-free measure.
Why It Matters
In machine learning, correlation helps identify which features move together, revealing potential redundancies or relationships. Before building models, understanding these linear relationships guides feature selection and helps detect multicollinearity.
Key Takeaway: Covariance measures how two variables vary together, while correlation standardizes this to a -1 to +1 scale, making linear relationships easy to interpret regardless of measurement units.