Course contentsShow
← Machine Learning and Deep Learning
Lesson 2292 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

KL Divergence as a Distance Metric

Using KL divergence to measure the distance between old and new policies in probability distribution space.

KL Divergence as a Distance Metric

What You'll Learn

You'll learn how KL divergence measures the "distance" between probability distributions, specifically how it quantifies the difference between old and new policy distributions in reinforcement learning.

The Core Idea

When updating policies in advanced RL algorithms, we need to ensure the new policy doesn't deviate too drastically from the old one. But how do we measure "distance" between two probability distributions? Enter Kullback-Leibler (KL) divergence.

KL divergence measures how one probability distribution differs from another. For policies π_old and π_new, the KL divergence is:

KL(π_old || π_new) = Σ π_old(a|s) * log(π_old(a|s) / π_new(a|s))

Think of it like this: imagine your old policy is a weather forecast predicting rain with certain probabilities. Your new policy is a slightly adjusted forecast. KL divergence tells you how "surprised" you'd be on average if events happened according to the old forecast but you believed the new one.

Why It Matters for Policy Updates

In trust region methods (building on the previous lesson), we use KL divergence as a constraint to keep policy updates conservative. Unlike simple Euclidean distance between parameters, KL divergence operates in probability space—it measures how the actual action distributions change, not just the network weights.

Important: KL divergence is not symmetric—KL(P || Q) ≠ KL(Q || P). This matters because we typically want to limit how much the new policy diverges from the old, not vice versa.

Key Takeaway: KL divergence quantifies the distance between probability distributions in a way that captures how different two policies are from each other's perspective, making it the natural metric for constraining policy updates in advanced RL algorithms.