Covariance and Correlation
What you'll learn: How to measure and interpret the linear relationship between two variables using covariance and correlation.
The Core Idea
Imagine you're tracking two variables—say, hours studied and test scores. Do they move together? When one increases, does the other tend to increase too? Covariance and correlation quantify this relationship.
Covariance measures how two variables change together. If both tend to be above their means at the same time (or below together), covariance is positive. If one is high when the other is low, it's negative. If they don't show a pattern, it's near zero.
The formula for covariance between variables X and Y:
Cov(X,Y) = E[(X - E[X])(Y - E[Y])]
However, covariance has a problem: its value depends on the scale of your variables. Studying hours in minutes vs. hours gives different covariances!
Pearson correlation coefficient solves this by normalizing covariance. It always falls between -1 and +1:
ρ(X,Y) = Cov(X,Y) / (σ_X × σ_Y)
Where σ represents standard deviation.
Interpretation
- ρ = +1: Perfect positive linear relationship
- ρ = 0: No linear relationship
- ρ = -1: Perfect negative linear relationship
- ρ = 0.7: Strong positive relationship
- ρ = -0.3: Weak negative relationship
Why It Matters in ML
Understanding which features are correlated helps you identify redundant information, detect patterns, and understand how variables influence each other before building models.
Key Takeaway: Covariance measures how two variables vary together, while correlation standardizes this to a -1 to +1 scale, making it easier to interpret the strength and direction of linear relationships.