From Linear Regression to Gaussian Processes
You'll learn how Gaussian Processes extend Bayesian linear regression from distributions over parameters to distributions over entire functions.
The Conceptual Leap
In Bayesian linear regression, you place a prior distribution over the model's weights (parameters), then update this to a posterior after seeing data. The predictions come from these weight distributions. But here's the shift: what if instead of putting probabilities on weights, we put probabilities directly on functions themselves?
A Gaussian Process (GP) does exactly that. Rather than saying "these weight values are probable," a GP says "these entire function shapes are probable." It's a distribution over infinitely many possible functions that could explain your data.
The Connection to Bayesian Linear Regression
Think of Bayesian linear regression with basis functions: you model y = w₁φ₁(x) + w₂φ₂(x) + ... with a prior on weights w. Each choice of w gives you one function. The collection of all possible functions (weighted by their probability under your prior and likelihood) forms an implicit distribution over functions.
Gaussian Processes make this explicit. They skip the intermediate weight representation and work directly with function properties. Any finite set of points follows a multivariate Gaussian distribution—hence "Gaussian Process."
The beauty: with the right prior and linear basis functions, Bayesian linear regression IS a Gaussian Process. GPs generalize this by allowing infinite basis functions through kernel functions.
Why This Matters
Instead of assuming a specific functional form (linear, polynomial), GPs let the data reveal the function's shape. You specify how smooth or wiggly functions should be (via the kernel), and the GP handles the rest—giving you both predictions and uncertainty estimates.
Key Takeaway: Gaussian Processes reframe Bayesian regression from "probable parameter values" to "probable function shapes," offering a flexible, principled way to model uncertainty about functions themselves.