Few-Shot Learning Benchmarks
What you'll learn: You'll discover the standard datasets used to fairly evaluate and compare few-shot learning methods.
Why We Need Benchmarks
Just as track and field athletes compete on standardized courses, few-shot learning methods need consistent testing grounds. Without shared benchmarks, it's impossible to know whether your Prototype Network genuinely outperforms a Matching Network or if one researcher just got lucky with easier data.
The Standard Benchmarks
miniImageNet
miniImageNet is the most popular few-shot learning benchmark. It's a subset of ImageNet containing:
- 100 classes (sampled from the full ImageNet dataset)
- 600 images per class (84×84 pixels, downsampled for efficiency)
- Typically split: 64 training classes, 16 validation, 20 test classes
Think of it as a "starter pack" from ImageNet—large enough to be challenging, small enough to train quickly during meta-training episodes.
tieredImageNet
tieredImageNet offers a more sophisticated split:
- 608 classes from ImageNet
- Organized hierarchically by ImageNet's semantic tree
- Training/validation/test classes come from different high-level categories
The key insight: tieredImageNet ensures training classes (e.g., "vehicles") don't overlap semantically with test classes (e.g., "animals"). This tests whether your model truly learns to learn, rather than memorizing visual features that happen to transfer.
Why These Benchmarks Matter
These datasets let researchers:
- Report standardized 5-way 1-shot and 5-way 5-shot accuracy
- Compare methods fairly (same support/query splits)
- Track progress in the field over time
Key Takeaway: miniImageNet and tieredImageNet provide standardized testing environments where few-shot methods compete on equal footing—miniImageNet offers accessible quick evaluation, while tieredImageNet tests true generalization across semantic boundaries.