Course contentsShow
← Machine Learning and Deep Learning
Lesson 1790 of 3,53838. Instruction Tuning and AlignmentFree lesson

The Policy Gradient Foundation

How policy gradients enable direct optimization of response quality through learned reward signals.

The Policy Gradient Foundation

What you'll learn: How policy gradients enable direct optimization of response quality through learned reward signals in RLHF.

The Core Problem

When fine-tuning language models with RLHF, we can't use traditional supervised learning because we don't have "correct" text outputs—we only have a reward model that scores responses. Policy gradients solve this by letting us optimize directly for higher rewards, even though the reward signal isn't differentiable in the usual way.

What Are Policy Gradients?

Think of your LLM as an agent choosing actions (tokens) based on a policy (the probability distribution over next tokens). Traditional backpropagation requires knowing the exact "right answer," but policy gradients work differently: they adjust the policy to make actions that led to high rewards more likely and actions that led to low rewards less likely.

The key insight: instead of computing gradients through the reward model itself, we use the reward as a weighting factor on the log-probability of the generated sequence. If a response gets a high reward, we increase the probability of generating similar responses. If it gets a low reward, we decrease it.

Mathematical Intuition

The policy gradient adjusts parameters θ to maximize expected reward:

∇θ E[reward] ≈ E[reward(y) × ∇θ log π_θ(y)]

This means: scale the gradient of the log-probability by how good the outcome was. No need to backpropagate through the reward model—just use its output as a multiplier.

Key Takeaway: Policy gradients transform the RLHF problem from "differentiate through a reward model" (impossible) to "weight likelihood updates by reward scores" (tractable), enabling direct optimization of LLM responses for human preferences.