Course contentsShow
← Machine Learning and Deep Learning
Lesson 366 of 3,5388. Unsupervised Learning: ClusteringFree lesson

Likelihood Function for GMMs

Writing the log-likelihood of observed data under a mixture model with latent cluster assignments.

Likelihood Function for GMMs

What you'll learn: How to mathematically express the probability that your observed data came from a Gaussian Mixture Model.

The Core Idea

Imagine you have a bag of data points, and you believe they came from several overlapping Gaussian distributions (clusters) mixed together. The likelihood function answers: "How likely is it that this specific mixture model generated the data I'm seeing?"

Since we don't know which cluster each point came from (these are latent or hidden assignments), we have to consider all possible clusters when calculating likelihood for each point.

Writing the Likelihood

For a single data point x, the likelihood under a mixture model is:

P(x | model) = π₁·N(x | μ₁, Σ₁) + π₂·N(x | μ₂, Σ₂) + ... + πₖ·N(x | μₖ, Σₖ)

Where:

  • πₖ = mixing weight (probability that a point belongs to cluster k)
  • N(x | μₖ, Σₖ) = Gaussian probability density for cluster k

For all data points X = {x₁, x₂, ..., xₙ}, the log-likelihood is:

log L(X) = Σᵢ log(Σₖ πₖ·N(xᵢ | μₖ, Σₖ))

We use log because it's easier to work with (turns products into sums) and is numerically more stable.

Analogy

Think of a voting poll where people don't declare their party. The likelihood asks: "Given what people said, how well does my model of '40% left-leaning, 60% right-leaning voters' explain the responses?" You sum over all possible party affiliations for each person.

Key Takeaway: The likelihood function for GMMs sums over all cluster contributions for each point, capturing uncertainty about latent cluster assignments; we typically maximize the log-likelihood to find the best model parameters.