Course contentsShow
← Machine Learning and Deep Learning
Lesson 743 of 3,53818. Regularization Techniques for Deep NetworksFree lesson

Dropout Rate Selection

Guidelines for choosing dropout probabilities for different layer types and network architectures.

Dropout Rate Selection

You'll learn: How to choose the right dropout probability for different layers and architectures to balance regularization and model capacity.

The Core Challenge

You know that dropout randomly "turns off" neurons during training to prevent overfitting. But how many neurons should you drop? Set the dropout rate too high, and your network loses too much capacity to learn. Set it too low, and you barely regularize at all.

The dropout rate (often called p) is the probability that a neuron's output will be zeroed during training.

General Guidelines

Think of dropout rates like seasoning in cooking—different dishes need different amounts:

Hidden Layers

  • Default starting point: 0.5 (drop 50% of neurons)
  • This is the "sweet spot" discovered empirically in many architectures
  • Provides strong regularization without crippling learning

Input Layers

  • Recommended range: 0.1 to 0.2 (drop only 10-20%)
  • Input features are precious—you don't want to discard too much raw information
  • Lower dropout preserves signal from the data

Output/Final Layers

  • Often: 0.0 (no dropout) or very low rates
  • These layers make final predictions and need stability
  • Exception: very large final layers may use modest dropout (0.2-0.3)

Architecture-Specific Adjustments

  • Deeper networks: May benefit from higher dropout (0.5-0.7) in middle layers
  • Smaller datasets: Consider higher dropout to prevent overfitting
  • Larger networks: More parameters mean more regularization might help

Key Takeaway: Start with 0.5 for hidden layers and 0.1-0.2 for input layers, then tune based on validation performance—if underfitting, lower the rate; if overfitting persists, increase it.