Course contentsShow
← Machine Learning and Deep Learning
Lesson 558 of 3,53814. Probabilistic and Bayesian Machine LearningFree lesson

Prior Distributions on Weights

Define prior beliefs over regression coefficients; explore common choices like Gaussian priors and their meanings.

Prior Distributions on Weights

What you'll learn: How to express your initial beliefs about model parameters before seeing any data, and why this matters in Bayesian machine learning.

The Core Idea

In Bayesian machine learning, we don't treat model weights (regression coefficients, neural network parameters) as fixed unknowns to estimate. Instead, we treat them as random variables with probability distributions. A prior distribution encodes what we believe about these weights before observing any training data.

Think of it like this: Imagine you're estimating someone's height. Even before measuring, you know it's probably between 4 and 7 feet—not 50 feet or negative. That's your "prior belief." Similarly, we can express beliefs about weights: "most weights are probably close to zero" or "weights shouldn't be extremely large."

Common Prior Choices

Gaussian (Normal) Prior

The most popular choice is a Gaussian prior centered at zero:

  • Meaning: We believe weights are likely small, with most mass near zero
  • Effect: Naturally encourages simpler models (similar to L2 regularization)
  • Example: Weight ~ Normal(0, σ²) means "I expect this weight near 0, with variance σ²"

A narrow Gaussian (small σ²) says "I strongly believe weights should be tiny." A wide Gaussian (large σ²) says "I'm less opinionated—weights can vary more."

Why Priors Matter

Priors let you inject domain knowledge: if you know certain features shouldn't have huge influence, use tighter priors on those weights. They also regularize your model, preventing overfitting by penalizing extreme parameter values.

Key Takeaway: Prior distributions on weights encode your initial beliefs about parameter values before seeing data; Gaussian priors centered at zero are most common, naturally encouraging smaller weights and simpler models.