Loss Functions and Objective Functions
You'll learn: How machine learning models mathematically measure prediction errors and use these measures to guide learning toward better performance.
What Are Loss and Objective Functions?
Imagine teaching a student archery. After each shot, you need a scoring system to tell them how far they missed the bullseye. Without that measurement, they can't improve systematically.
In machine learning, loss functions (also called cost functions) are precisely that scoring system. A loss function takes your model's predictions and compares them to the true labels, outputting a single number that quantifies how wrong the predictions are. Lower loss means better predictions.
The objective function is what your learning algorithm tries to optimize—usually minimizing the total loss across all training data. Think of it as the overall goal: "make the average miss as small as possible across all arrows."
Why They Matter
Your choice of loss function shapes everything your model learns. Different losses penalize different types of errors:
- Mean Squared Error (MSE): Heavily penalizes large mistakes (like being way off the bullseye)
- Absolute Error: Treats all miss distances more uniformly
- Cross-Entropy Loss: Measures how well predicted probabilities match true class labels
The hypothesis space defines what models are possible, but the loss function defines what makes one hypothesis better than another. During training, the algorithm searches the hypothesis space, using the objective function as a compass pointing toward better models.
Key Takeaway: Loss functions translate the abstract goal of "good predictions" into concrete numbers that algorithms can minimize, transforming learning from intuition into mathematical optimization guided by measured prediction error.