Gibbs Sampling for Conditional Distributions
You'll learn: How Gibbs sampling simplifies multivariate inference by breaking it into easier one-variable-at-a-time updates using conditional distributions.
The Core Idea
Gibbs sampling is a specialized MCMC technique that makes sampling from complex joint distributions much simpler. Instead of proposing jumps in all dimensions at once (like Metropolis-Hastings), Gibbs sampling cycles through variables one at a time, sampling each from its conditional distribution given the current values of all others.
Think of it like updating a group photo where everyone must hold still except the one person being repositioned. You adjust one person's pose based on where everyone else is currently standing, then move to the next person, and cycle through repeatedly.
How It Works
For a joint distribution over variables (θ₁, θ₂, ..., θₙ):
- Initialize all variables with starting values
- Sample θ₁ from P(θ₁ | θ₂, θ₃, ..., θₙ) — conditioning on all others
- Sample θ₂ from P(θ₂ | θ₁, θ₃, ..., θₙ) — using the new θ₁
- Continue for each variable
- Repeat the full cycle many times
Each complete cycle produces one sample from the joint distribution (after burn-in).
Why Gibbs Sampling Shines
The magic happens when conditional distributions are much simpler than the joint distribution—often with closed forms. This is especially true with conjugate priors in Bayesian models. No acceptance/rejection step is needed; every sample is kept.
Key Takeaway: Gibbs sampling transforms hard multivariate sampling into a sequence of easier univariate conditional samples, cycling through variables systematically to explore the joint distribution.