Evaluation Metrics for Collaborative Filtering
What you'll learn: How to properly measure whether your recommender system is actually making good predictions and useful recommendations.
Why Metrics Matter
Building a collaborative filtering model is only half the battle—you need to know if it's working well. Different metrics capture different aspects of recommendation quality, from prediction accuracy to ranking relevance.
Core Evaluation Metrics
RMSE (Root Mean Squared Error)
When your system predicts ratings (like "this user will rate this movie 4.2 stars"), RMSE measures how far off your predictions are from actual ratings. Lower is better. It penalizes large errors more heavily than small ones.
Think of it like measuring how badly you missed the bullseye—missing by 2 points counts more than twice as bad as missing by 1 point.
Precision@k and Recall@k
These metrics evaluate your top-k recommendations (e.g., "top 10 items to recommend"):
- Precision@k: Of the k items you recommended, how many were actually relevant? (Quality focus)
- Recall@k: Of all relevant items, how many did you manage to include in your top-k? (Coverage focus)
Imagine recommending 10 movies: if 7 are good (precision@10 = 0.7) but there were 20 good movies total (recall@10 = 7/20 = 0.35), you have high precision but lower recall.
NDCG (Normalized Discounted Cumulative Gain)
Ranking metrics like NDCG evaluate whether you're putting the most relevant items at the top of your list. Getting the order right matters—users rarely scroll past the first few recommendations. NDCG ranges from 0 to 1, with 1 being perfect ranking.
When to Use What
Use RMSE when explicit ratings matter. Use precision/recall when you care about identifying good items. Use NDCG when ranking order is critical.
Key Takeaway: Choose metrics that match your business goal—RMSE for rating prediction accuracy, precision@k/recall@k for recommendation relevance, and NDCG for ranking quality.