Course contentsShow
← Machine Learning and Deep Learning
Lesson 367 of 3,5388. Unsupervised Learning: ClusteringFree lesson

The Expectation-Maximization Algorithm

Overview of EM as an iterative two-step procedure for fitting models with latent variables.

The Expectation-Maximization Algorithm

What you'll learn: How EM provides an iterative method to fit probabilistic models when some variables are hidden or unobserved.

The Challenge with Hidden Variables

When working with Gaussian Mixture Models, we face a chicken-and-egg problem: to estimate the Gaussian parameters (means, covariances), we need to know which cluster each point belongs to. But to assign points to clusters, we need to know the Gaussian parameters! This is where Expectation-Maximization (EM) comes to the rescue.

The Two-Step Dance

EM is an iterative algorithm that alternates between two steps:

E-Step (Expectation)

Given current parameter estimates, compute the probability that each data point belongs to each cluster. These are called "responsibilities" — soft assignments rather than hard cluster labels. Think of it as asking: "Given what I currently believe about my clusters, how likely is it that this point came from cluster 1? Cluster 2?"

M-Step (Maximization)

Using these probability assignments, update the model parameters (means, covariances, mixing weights) to maximize the likelihood. It's like saying: "Now that I have soft assignments, what cluster parameters best explain my data?"

The Iteration

You repeat E-step → M-step → E-step → M-step until convergence (when parameters stop changing significantly). Each iteration is guaranteed to improve or maintain the likelihood function — you never move backward.

Analogy: Imagine sorting colored marbles in dim light. First, you guess colors (E-step), then adjust your color definitions based on those guesses (M-step). With each round, your color model and sorting improve together.

Key Takeaway: The Expectation-Maximization algorithm solves the latent variable problem by iteratively alternating between computing soft cluster assignments (E-step) and updating model parameters (M-step) until convergence.