Weighted KNN and Distance Weighting
What You'll Learn
You'll discover how to make KNN smarter by giving closer neighbors more say in predictions, especially near decision boundaries.
The Core Idea
In standard KNN, all K neighbors get an equal vote—whether they're right next door or relatively far away. Weighted KNN improves this by giving closer neighbors more influence using inverse distance weighting.
Think of asking friends for restaurant recommendations. You'd probably trust your neighbor who lives on the same street more than someone from across town, even if both went to the restaurant. That's exactly what weighted KNN does with data points.
How Inverse Distance Weighting Works
Instead of each neighbor contributing equally (weight = 1), each neighbor's contribution is proportional to how close it is:
Weight = 1 / distance
or sometimes:
Weight = 1 / distance²
This means:
- A neighbor at distance 0.5 has weight 2 (or 4 with squared distance)
- A neighbor at distance 2 has weight 0.5 (or 0.25 with squared distance)
- Closer neighbors dominate the prediction
Why This Matters
Weighted KNN shines at decision boundaries—those tricky regions where classes meet. Without weighting, a test point near the boundary might be swayed by distant neighbors from the wrong class. With distance weighting, nearby points (which are more likely to share the same class) have stronger influence, leading to cleaner, more accurate boundaries.
This is especially valuable when K is larger, preventing faraway outliers from diluting the signal from your true nearest neighbors.
Key Takeaway: Weighted KNN uses inverse distance to give closer neighbors more voting power, improving prediction accuracy especially at class boundaries where standard KNN struggles.