Swish and SiLU: Self-Gated Activations
What you'll learn: How Swish and SiLU activation functions use a sigmoid "gate" to create smooth, self-modulating nonlinearities that often outperform ReLU.
The Core Idea
Unlike ReLU which abruptly cuts off negative values, Swish (also called SiLU - Sigmoid Linear Unit) creates a smooth, self-gated activation by multiplying the input by its own sigmoid. The formula is elegantly simple:
Swish(x) = x · σ(x) where σ(x) is the sigmoid function.
Think of it as the input asking itself: "How much of me should I let through?" When x is large and positive, sigmoid ≈ 1, so you get approximately x (like ReLU). When x is large and negative, sigmoid ≈ 0, so the output approaches zero. But crucially, for values near zero, the function is smooth and non-monotonic — it can actually dip slightly below zero before recovering.
Why "Self-Gated"?
The sigmoid acts as a gate that modulates the input based on the input's own value. Unlike Leaky ReLU where the slope is fixed, or PReLU where it's learned globally, Swish adjusts its behavior dynamically based on each individual activation value. This self-gating provides adaptive behavior without extra parameters.
Key Properties
- Smooth: Infinitely differentiable (no sharp corners like ReLU)
- Non-monotonic: Can decrease slightly for negative values before rising
- Unbounded above: Like ReLU, grows linearly for large positive inputs
- Bounded below: Approaches zero (not negative infinity)
This smoothness helps gradients flow better during training, while the non-monotonic behavior provides additional expressiveness that can improve model performance, particularly in deep networks.
Key Takeaway: Swish/SiLU activations multiply the input by its sigmoid, creating a smooth, self-gating function that adapts to each value and often trains more effectively than ReLU.