Dropout During Training vs Inference
What you'll learn: How dropout behaves differently during training versus test time, and why this distinction is critical for proper model evaluation.
The Core Distinction
When you apply dropout during training, you randomly "turn off" a fraction of neurons (set their outputs to zero) with probability p. This forces the network to learn redundant representations and prevents over-reliance on specific neurons.
But here's the crucial part: dropout is only active during training. At inference (test time), you want your model to use its full capacity and make the most reliable predictions possible.
Why Turn Dropout Off at Test Time?
Imagine you're assembling a team presentation. During practice (training), you randomly exclude team members so everyone learns to cover for each other. But during the actual presentation (inference), you want everyone present to deliver the best result.
If you kept dropout active during inference, your predictions would be:
- Stochastic (different each time you run the same input)
- Weaker (using only a subset of the network's learned knowledge)
Scaling Activations
When dropout with probability p is disabled at test time, each neuron suddenly receives input from all its predecessors instead of just (1-p) fraction. To compensate, activations are scaled:
- Training: Multiply active neurons by
1/(1-p)(inverse dropout) - Inference: Use all neurons without scaling
This ensures the expected sum of inputs remains consistent between training and testing.
In Practice
Modern frameworks handle this automatically through "training mode" vs "evaluation mode" — dropout layers know when to activate or deactivate themselves.
Key Takeaway: Dropout is a training-time regularization technique that must be disabled during inference to ensure deterministic, full-strength predictions. Proper scaling ensures consistent activation magnitudes across both phases.