Course contentsShow
← Machine Learning and Deep Learning
Lesson 567 of 3,53814. Probabilistic and Bayesian Machine LearningFree lesson

From Linear Regression to Gaussian Processes

Understanding GPs as distributions over functions rather than parameters, and the connection to Bayesian linear regression.

From Linear Regression to Gaussian Processes

You'll learn how Gaussian Processes extend Bayesian linear regression from distributions over parameters to distributions over entire functions.

The Conceptual Leap

In Bayesian linear regression, you place a prior distribution over the model's weights (parameters), then update this to a posterior after seeing data. The predictions come from these weight distributions. But here's the shift: what if instead of putting probabilities on weights, we put probabilities directly on functions themselves?

A Gaussian Process (GP) does exactly that. Rather than saying "these weight values are probable," a GP says "these entire function shapes are probable." It's a distribution over infinitely many possible functions that could explain your data.

The Connection to Bayesian Linear Regression

Think of Bayesian linear regression with basis functions: you model y = w₁φ₁(x) + w₂φ₂(x) + ... with a prior on weights w. Each choice of w gives you one function. The collection of all possible functions (weighted by their probability under your prior and likelihood) forms an implicit distribution over functions.

Gaussian Processes make this explicit. They skip the intermediate weight representation and work directly with function properties. Any finite set of points follows a multivariate Gaussian distribution—hence "Gaussian Process."

The beauty: with the right prior and linear basis functions, Bayesian linear regression IS a Gaussian Process. GPs generalize this by allowing infinite basis functions through kernel functions.

Why This Matters

Instead of assuming a specific functional form (linear, polynomial), GPs let the data reveal the function's shape. You specify how smooth or wiggly functions should be (via the kernel), and the GP handles the rest—giving you both predictions and uncertainty estimates.

Key Takeaway: Gaussian Processes reframe Bayesian regression from "probable parameter values" to "probable function shapes," offering a flexible, principled way to model uncertainty about functions themselves.