Support Vectors
What you'll learn: You'll discover which specific data points from your training set are the true decision-makers in a Support Vector Machine.
The Critical Few
Imagine you're building a fence to separate two groups of animals in a field. Most animals could wander far from the fence without affecting where you place it. But a few animals stand right at the boundary—if they moved even slightly, you'd need to reposition the entire fence.
Support vectors are exactly those critical data points.
What Makes a Data Point a Support Vector?
In the hard-margin SVM objective you just learned about, the goal is to maximize the margin—the distance between the decision boundary and the nearest data points from each class.
Support vectors are:
- The data points that lie exactly on the margin boundaries (the two parallel lines on either side of the decision boundary)
- The closest points from each class to the decision boundary
- The only points that actually define where the decision boundary sits
Here's the remarkable part: you could remove all other training points, keep only the support vectors, and you'd get the exact same decision boundary. Every other point is too far away to matter—they're already correctly classified with room to spare.
Why "Support" Vectors?
These points literally "support" or hold up the decision boundary. They're under maximum tension—move one support vector, and the entire hyperplane shifts. The decision boundary is mathematically determined solely by these critical points.
Key Takeaway: Support vectors are the data points sitting on the margin boundaries—they alone determine the decision boundary, making them the most important points in your entire training set for SVM classification.