Course contentsShow
← Machine Learning and Deep Learning
Lesson 52 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Numerical Differentiation

Approximating derivatives computationally using finite differences when analytical solutions are unavailable.

Numerical Differentiation

What you'll learn: How to approximate derivatives using computers when you can't find them by hand.

The Core Idea

You've learned how to find derivatives analytically—using rules like the power rule, chain rule, and so on. But what happens when you have a function that's too complex, unknown, or just messy to differentiate by hand? That's where numerical differentiation comes in.

The fundamental concept is surprisingly simple: remember the derivative definition you learned earlier? It's the limit of a difference quotient as the step size approaches zero. Numerical differentiation uses that same idea, but stops before reaching the limit—it picks a small (but not zero) step size and computes an approximation.

The Finite Difference Method

The most common approach is the finite difference. Imagine you have a function f(x) and want to approximate its derivative at point x:

Forward difference:

f'(x) ≈ [f(x + h) - f(x)] / h

Here, h is a small number (like 0.001). You're essentially asking: "How much does f change over a tiny step forward?"

Central difference (often more accurate):

f'(x) ≈ [f(x + h) - f(x - h)] / (2h)

This averages the forward and backward changes, reducing error.

Why It Matters in ML

In machine learning, you often need gradients of complex loss functions to train models. Sometimes these functions involve simulations, black-box operations, or external systems where analytical derivatives are impossible. Numerical differentiation lets you still compute gradients—though it's slower and less precise than analytical methods.

Key Takeaway: Numerical differentiation approximates derivatives by evaluating a function at nearby points with small step sizes, providing a practical alternative when analytical differentiation isn't feasible.