Evaluation Protocols for Metric Learning
What you'll learn: How to properly construct evaluation episodes and validate few-shot metric learning models using N-way K-shot protocols.
The Core Challenge
When you've trained a metric learning model (like Siamese Networks, Prototypical Networks, or Matching Networks), how do you know it actually works? You can't just use standard accuracy on a fixed test set—few-shot learning is about generalizing to new classes never seen during training.
N-Way K-Shot Episode Construction
An evaluation episode mirrors your training setup but uses entirely different classes from your meta-testing split:
- Sample N classes randomly from held-out test classes
- Sample K examples per class for the support set (to learn from)
- Sample additional examples from those same N classes as queries (to predict)
- Compute accuracy on the query set after using only the K-shot support
For example, in 5-way 1-shot evaluation, you randomly pick 5 test classes, show the model 1 example of each, then ask it to classify 15 new images from those same 5 classes.
Proper Validation Strategy
Run many episodes (typically 600-10,000) and report:
- Mean accuracy across all episodes
- 95% confidence intervals (since each episode is random)
- Consistent N-way and K-shot settings (e.g., 5-way 1-shot AND 5-way 5-shot)
Think of it like a pop quiz where the teacher keeps changing which 5 topics you're tested on—you must prove your learning strategy works, not just memorization.
Why This Matters
Standard validation (one fixed test set) would let your model memorize test class embeddings. Episode-based evaluation ensures your distance metric truly generalizes to unseen categories, testing the model's ability to learn from minimal examples repeatedly.
Key Takeaway: Evaluate metric learning models using many N-way K-shot episodes from held-out classes, reporting mean accuracy and confidence intervals to validate true few-shot generalization capability.