Derivatives of Exponential Functions
What you'll learn: How to find the derivative of exponential functions, especially e^x, which appears constantly in neural network activation functions.
Why Exponentials Matter in ML
Exponential functions pop up everywhere in machine learning—sigmoid activations, softmax outputs, and probability distributions all rely on e^x. To train neural networks effectively, you need to compute gradients, which means taking derivatives of these exponential functions.
The Core Idea
The exponential function e^x has a beautiful mathematical property: its derivative is itself.
If f(x) = e^x, then f'(x) = e^x
This means the rate of change of e^x at any point equals the function's value at that point. If e^2 ≈ 7.39, then the slope of the curve at x=2 is also 7.39.
Why This Happens
Think of exponential growth like compound interest: the more you have, the faster it grows. The derivative measures "how fast," and for e^x, that growth rate is proportional to the current amount—they're exactly equal.
Chain Rule Extension
When you have e^(something), use the chain rule you learned earlier:
If f(x) = e^(g(x)), then f'(x) = e^(g(x)) × g'(x)
Example: For f(x) = e^(2x):
- Outer function: e^u where u = 2x
- Inner function derivative: g'(x) = 2
- Result: f'(x) = e^(2x) × 2 = 2e^(2x)
Other Exponential Bases
For a general base a^x (like 2^x or 10^x):
- Derivative: a^x × ln(a)
The natural base e is special because ln(e) = 1, making the derivative simplest.
Key Takeaway: The derivative of e^x is e^x itself—this unique property makes exponential functions computationally elegant in neural networks, where you'll repeatedly compute gradients during backpropagation.