Likelihood Function for GMMs
What you'll learn: How to mathematically express the probability that your observed data came from a Gaussian Mixture Model.
The Core Idea
Imagine you have a bag of data points, and you believe they came from several overlapping Gaussian distributions (clusters) mixed together. The likelihood function answers: "How likely is it that this specific mixture model generated the data I'm seeing?"
Since we don't know which cluster each point came from (these are latent or hidden assignments), we have to consider all possible clusters when calculating likelihood for each point.
Writing the Likelihood
For a single data point x, the likelihood under a mixture model is:
P(x | model) = π₁·N(x | μ₁, Σ₁) + π₂·N(x | μ₂, Σ₂) + ... + πₖ·N(x | μₖ, Σₖ)
Where:
- πₖ = mixing weight (probability that a point belongs to cluster k)
- N(x | μₖ, Σₖ) = Gaussian probability density for cluster k
For all data points X = {x₁, x₂, ..., xₙ}, the log-likelihood is:
log L(X) = Σᵢ log(Σₖ πₖ·N(xᵢ | μₖ, Σₖ))
We use log because it's easier to work with (turns products into sums) and is numerically more stable.
Analogy
Think of a voting poll where people don't declare their party. The likelihood asks: "Given what people said, how well does my model of '40% left-leaning, 60% right-leaning voters' explain the responses?" You sum over all possible party affiliations for each person.
Key Takeaway: The likelihood function for GMMs sums over all cluster contributions for each point, capturing uncertainty about latent cluster assignments; we typically maximize the log-likelihood to find the best model parameters.