Course contentsShow
← Machine Learning and Deep Learning
Lesson 500 of 3,53812. Model Validation and TuningFree lesson

Cross-Validation for Small Datasets

Adapting CV strategies when data is limited, including repeated k-fold and bootstrap methods.

Cross-Validation for Small Datasets

What you'll learn: How to reliably validate models when you have very limited data, using specialized cross-validation strategies.

Why Small Datasets Need Special Attention

When your dataset is tiny (think dozens or low hundreds of samples), a standard train-test split wastes precious data. Even basic k-fold cross-validation might not give stable estimates because each fold contains very few examples. You need strategies that extract maximum information from minimal data.

Repeated K-Fold Cross-Validation

Instead of running k-fold CV once, you repeat it multiple times with different random shuffles. Each repetition creates new fold assignments, giving you more performance measurements to average.

Example: Run 5-fold CV ten times (10 repetitions). You get 50 total scores instead of 5, producing a more stable and trustworthy estimate of model performance. This reduces the variance caused by unlucky fold splits.

Bootstrap Methods

Bootstrapping creates multiple training sets by sampling your data with replacement. Some samples appear multiple times, others not at all. The unused samples (roughly 37% on average) serve as a test set.

This approach generates many slightly different datasets from your limited pool, letting you assess how your model performs across varied training scenarios—crucial when every data point counts.

The Trade-Off

Both methods increase computational cost (you're training many more models), but when data is scarce, that's a worthy investment. The extra iterations provide statistical confidence that your single precious dataset can't offer alone.

Key Takeaway: When working with small datasets, use repeated k-fold cross-validation or bootstrap methods to extract maximum validation signal from minimal data, trading computation time for reliability.