Prior Distributions on Weights
What you'll learn: How to express your initial beliefs about model parameters before seeing any data, and why this matters in Bayesian machine learning.
The Core Idea
In Bayesian machine learning, we don't treat model weights (regression coefficients, neural network parameters) as fixed unknowns to estimate. Instead, we treat them as random variables with probability distributions. A prior distribution encodes what we believe about these weights before observing any training data.
Think of it like this: Imagine you're estimating someone's height. Even before measuring, you know it's probably between 4 and 7 feet—not 50 feet or negative. That's your "prior belief." Similarly, we can express beliefs about weights: "most weights are probably close to zero" or "weights shouldn't be extremely large."
Common Prior Choices
Gaussian (Normal) Prior
The most popular choice is a Gaussian prior centered at zero:
- Meaning: We believe weights are likely small, with most mass near zero
- Effect: Naturally encourages simpler models (similar to L2 regularization)
- Example: Weight ~ Normal(0, σ²) means "I expect this weight near 0, with variance σ²"
A narrow Gaussian (small σ²) says "I strongly believe weights should be tiny." A wide Gaussian (large σ²) says "I'm less opinionated—weights can vary more."
Why Priors Matter
Priors let you inject domain knowledge: if you know certain features shouldn't have huge influence, use tighter priors on those weights. They also regularize your model, preventing overfitting by penalizing extreme parameter values.
Key Takeaway: Prior distributions on weights encode your initial beliefs about parameter values before seeing data; Gaussian priors centered at zero are most common, naturally encouraging smaller weights and simpler models.