Course contentsShow
← Machine Learning and Deep Learning
Lesson 577 of 3,53814. Probabilistic and Bayesian Machine LearningFree lesson

GPs for Classification

Extending GPs to binary and multiclass classification using the probit or logistic likelihood.

GPs for Classification

What You'll Learn

You'll learn how to extend Gaussian Processes from regression to classification tasks by using specialized likelihood functions that handle discrete class labels.


The Core Challenge

Gaussian Processes naturally output continuous values with uncertainty, but classification requires discrete predictions (e.g., "spam" or "not spam"). The trick is to map GP outputs through a link function that converts continuous values into probabilities, then use those probabilities with a likelihood function for binary or multiclass labels.


Binary Classification with GPs

For binary classification, we place a GP prior over a latent function f(x) that represents an underlying continuous "decision score." We then squash this through:

  • Probit link: Uses the cumulative Gaussian function Φ(f(x)) to get P(y=1|x)
  • Logistic link: Uses the sigmoid σ(f(x)) = 1/(1+e^(-f(x)))

The likelihood becomes Bernoulli: P(y|f(x)) where y ∈ {0,1}.

Unlike regression, there's no closed-form posterior. You must use approximation methods like Laplace approximation or variational inference to estimate the posterior over f(x), then make predictions by integrating over this uncertainty.


Multiclass Extension

For K classes, you can use K independent GPs (one-vs-rest) or a single multi-output GP with a softmax link to produce class probabilities. Each approach balances computational cost against modeling class correlations.


Key Takeaway:
GPs extend to classification by modeling a latent continuous function and mapping it through probit or logistic links, requiring approximation methods since the posterior is no longer analytically tractable.