PPO Training Loop Architecture
What You'll Learn
You'll learn how to orchestrate the four essential models in PPO training—policy, reference, reward, and value networks—into a cohesive training pipeline for RLHF.
The Four-Model Orchestra
Think of PPO training like conducting a symphony with four distinct instruments, each playing a specific role:
1. Policy Network (The Lead Performer)
This is your trainable LLM that generates responses. It's the only model whose weights update during PPO training. The policy network produces text completions and improves based on rewards.
2. Reference Network (The Anchor)
A frozen copy of your initial SFT model. It stays constant throughout training, providing a stable baseline to compute the KL divergence penalty. This prevents your policy from drifting too far from reasonable behavior.
3. Reward Model (The Judge)
A frozen network that scores generated text. For each policy output, it provides a scalar reward signal indicating quality, helpfulness, or alignment. This guides optimization but never updates.
4. Value Network (The Predictor)
Estimates expected future rewards for advantage calculation. Often shares architecture with the policy (sometimes sharing layers). This model trains alongside the policy to improve advantage estimates.
The Training Pipeline Flow
- Generate rollouts: Policy produces text completions
- Score with reward model: Get reward signals for each completion
- Compute advantages: Value network predicts expected returns; compare with actual rewards
- Calculate KL penalty: Reference network measures divergence from original policy
- Update policy and value: Use clipped surrogate objective with multiple mini-batch epochs
- Repeat: Continue until convergence
The reference model acts as your safety net, the reward model as your compass, the value network as your fortune teller, and the policy as your evolving student.
Key Takeaway: PPO orchestrates four models—trainable policy and value networks, frozen reference and reward models—in a structured loop that generates text, scores it, computes advantages with KL constraints, and updates the policy iteratively.