Fisher Information Matrix
What you'll learn: How the Fisher Information Matrix serves as a natural metric for measuring distances in policy space and enables safe policy updates in TRPO.
What Is the Fisher Information Matrix?
The Fisher Information Matrix (FIM) measures how much information a distribution contains about its parameters. In deep RL, specifically in TRPO, it tells us how sensitive our policy's output probabilities are to changes in the neural network's parameters.
Think of it like this: imagine you're walking on a hilly landscape where each point represents a different policy. The Fisher Information Matrix describes the "curvature" of this landscape. In some directions, a small step in parameter space causes huge changes in policy behavior (steep); in others, barely anything changes (flat).
Why It Matters for TRPO
Remember how TRPO uses KL divergence to measure the distance between old and new policies? The Fisher Information Matrix provides a local approximation of this KL divergence. Specifically, for small changes:
KL(π_old || π_new) ≈ ½ δᵀ F δ
where δ is the change in parameters and F is the Fisher Information Matrix.
This quadratic approximation transforms the trust region constraint into a more computationally tractable form. Instead of directly computing KL divergence (expensive!), we can use the FIM to approximate it, which is exactly what the conjugate gradient method operates on when solving TRPO's constrained optimization problem.
The Natural Metric
The FIM defines a "natural" geometry for policy space because it's invariant to reparameterization—the true distance between two policies doesn't change just because you represent them differently.
Key Takeaway: The Fisher Information Matrix acts as a natural ruler in policy space, enabling TRPO to measure how much a policy will actually change from a parameter update, ensuring stable, trust-region-constrained learning.