Parametric ReLU (PReLU): Learning the Slope
What you'll learn: How PReLU turns the negative slope into a trainable parameter that the network learns during backpropagation.
The Problem with Fixed Slopes
You've seen Leaky ReLU, which fixes the "dying ReLU" problem by allowing a small negative slope (typically 0.01) for inputs below zero. But here's the question: who says 0.01 is the best value? What if different neurons in different layers need different slopes?
Making the Slope Learnable
Parametric ReLU (PReLU) takes Leaky ReLU one step further: instead of hardcoding the negative slope, it treats the slope as a learnable parameter that updates during training via backpropagation.
The PReLU function is:
f(x) = x if x > 0
f(x) = α·x if x ≤ 0
The key difference: α is not fixed — it starts with an initial value (often 0.25) and the network learns its optimal value through gradient descent, just like it learns weights and biases.
Why This Matters
Think of PReLU as giving each neuron (or channel) the freedom to choose its own "personality" for handling negative inputs. One neuron might learn α = 0.01 (nearly flat), while another might learn α = 0.3 (steeper slope), depending on what helps minimize the loss function.
This adaptive behavior means:
- No manual tuning of the slope hyperparameter
- Different parts of the network can specialize differently
- The network automatically balances between ReLU-like and Leaky ReLU-like behavior
Key Takeaway: PReLU transforms the negative slope from a fixed hyperparameter into a learnable parameter, allowing the network to adaptively determine the best activation behavior for each neuron during training.