Independence of Random Variables
What you'll learn: How to recognize and test whether two random variables are statistically independent—a crucial concept for understanding data relationships in ML.
Core Idea
Two random variables are independent if knowing the value of one gives you no information about the other. Think of it like flipping a coin and rolling a die: the coin result doesn't affect the die, and vice versa.
Formally, random variables X and Y are independent if:
P(X = x, Y = y) = P(X = x) · P(Y = y)
In words: the joint probability equals the product of their individual (marginal) probabilities for all possible values of x and y.
Testing for Independence
Using Probability Distributions
For discrete variables, check if the joint PMF factorizes:
- If p(x, y) = p(x) · p(y) everywhere → independent
- If this fails for even one pair → not independent
For continuous variables, check if the joint PDF factorizes:
- If f(x, y) = f(x) · f(y) everywhere → independent
Using Covariance
Remember from the previous lesson that covariance measures linear relationship. If X and Y are independent, then Cov(X, Y) = 0.
⚠️ Warning: The reverse isn't always true! Zero covariance doesn't guarantee independence (it only rules out linear dependence).
Intuition
Independence is stronger than zero correlation. Independent variables have no relationship whatsoever—not linear, not nonlinear, nothing. Zero covariance just means no linear pattern.
Key Takeaway: Random variables X and Y are independent when their joint probability always equals the product of their marginals, meaning knowing one reveals nothing about the other—a foundational assumption in many ML algorithms.