Course contentsShow
← Computer Science
Lesson 2801 of 2,87262. Machine Learning FundamentalsFree lesson

Holdout Method Limitations

Understanding variance in single train-test splits and when simple holdout is insufficient.

Holdout Method Limitations

You'll learn why splitting data once into train and test sets can give unreliable performance estimates, especially with small datasets.

The Problem with a Single Split

The holdout method (splitting data once into train and test) is simple and fast, but it introduces high variance in your performance estimate. Your model's reported accuracy depends heavily on which specific examples ended up in the test set.

Imagine you have 100 data points. If you randomly select 20 for testing, you might get unlucky and pick the 20 hardest examples—your model looks terrible. Split again with different randomness, and you might pick 20 easy ones—suddenly your model looks great! The model didn't change; only the split did.

When Simple Holdout Fails

Small datasets are especially vulnerable. With only 200 examples and a 20% test split, your test set has just 40 points. A few unusual examples can swing your accuracy by several percentage points. The smaller your dataset, the noisier your single estimate becomes.

Imbalanced classes amplify the problem. If you're detecting rare fraud (1% positive rate), a random split might leave your test set with zero fraud cases—rendering evaluation meaningless.

High-variance models (like deep neural networks or complex trees) can appear to perform very differently depending on the split, making it hard to know their true capability.

The Solution Foreshadowed

This is why you've learned about k-fold cross-validation and stratified sampling—they reduce this variance by averaging over multiple splits or ensuring representative distributions. Simple holdout is fine for huge datasets or quick prototyping, but rigorous evaluation demands more stable methods.

Key Takeaway: A single train-test split produces high-variance performance estimates; with small or imbalanced data, one unlucky split can mislead you about your model's true quality.