First-Order MAML (FOMAML)
What you'll learn: How to make MAML computationally practical by simplifying its gradient calculations without sacrificing much performance.
The Computational Challenge
MAML's power comes from computing second-order gradients—gradients of gradients. When you update parameters in the inner loop and then compute outer loop gradients, you're technically differentiating through the entire inner optimization process. This means calculating how the outer loss changes with respect to initial parameters, through the adapted parameters. While elegant, this requires computing and storing Hessian matrices (second derivatives), which is memory-intensive and slow.
The First-Order Approximation
First-Order MAML (FOMAML) makes a clever simplification: it treats the inner loop's adapted parameters as constants when computing outer loop gradients. Instead of backpropagating through the entire inner loop optimization, FOMAML only uses first-order gradients—just like standard gradient descent.
Think of it like this: Imagine you're learning to adjust a recipe (meta-learning). Full MAML asks "how should I change my starting recipe considering how every adjustment ripples through my cooking process?" FOMAML simplifies to "how should I change my starting recipe based on the final dish, ignoring the detailed cooking dynamics?"
Why It Works
Surprisingly, FOMAML retains most of MAML's effectiveness. The inner loop adaptation still happens normally—you still fine-tune on the support set. You're only simplifying the meta-gradient computation. This dramatically reduces memory usage and computation time (often 30-50% faster) while maintaining strong few-shot performance.
The trade-off: slightly less precise meta-gradients, but the practical gains often outweigh this theoretical loss.
Key Takeaway: FOMAML approximates MAML by ignoring second-order derivatives during meta-updates, making meta-learning significantly faster and more memory-efficient while maintaining competitive few-shot learning performance.