Rollout Generation and Experience Collection
What you'll learn: How to generate trajectory samples from your current policy and gather the reward signals needed to train with PPO.
The Core Idea
In RLHF with PPO, you can't just train on static datasets—you need fresh experiences from your evolving policy. Rollout generation means sampling complete response sequences from your current language model (the policy) given various prompts, then collecting the rewards for each of those generations. This creates the training data for your next policy update.
Think of it like a student learning to write essays: they submit drafts (rollouts), receive grades (rewards), and use that feedback to improve. But as they get better, they need new drafts evaluated—old feedback becomes stale.
The Process
- Sample prompts from your instruction dataset
- Generate responses by sampling from your current policy π_θ (your LLM with current weights)
- Compute rewards for each (prompt, response) pair using your trained reward model
- Store the experience: save the prompt, generated tokens, log probabilities, and rewards
- Use for training: this batch of rollouts becomes your training data for the PPO update
The key is that you're sampling from the current policy, not replaying old data. As your model improves through PPO updates, you generate new rollouts to stay aligned with the policy's current behavior.
Why Fresh Rollouts Matter
Your value function and advantage estimates depend on what the current policy would do. Using outdated rollouts would give misleading gradients because the policy has changed. This is called the "on-policy" requirement—PPO needs recent experience from the policy being optimized.
Key Takeaway: Rollout generation creates training experience by sampling complete responses from your current policy and collecting their rewards, ensuring PPO trains on up-to-date policy behavior rather than stale data.