Course contentsShow
← Machine Learning and Deep Learning
Lesson 2287 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Off-Policy Actor-Critic: ACER and SAC Preview

Introduce off-policy actor-critic methods that use experience replay for sample efficiency.

Off-Policy Actor-Critic: ACER and SAC Preview

What You'll Learn

You'll discover how off-policy actor-critic methods leverage experience replay to dramatically improve sample efficiency beyond traditional on-policy approaches like A3C.

The Sample Efficiency Challenge

Traditional actor-critic methods like A3C are on-policy: they learn only from fresh experiences collected by the current policy. Once you update your policy, those experiences become "stale" and must be discarded. This is wasteful—imagine throwing away every practice problem after taking one quiz!

Off-Policy Actor-Critic: Learning from the Past

Off-policy actor-critic methods break this limitation by incorporating experience replay (which you've seen in DQN) into the actor-critic framework. The key insight: store past experiences in a replay buffer and reuse them for learning, even after the policy has changed.

Two prominent algorithms:

  • ACER (Actor-Critic with Experience Replay): Adapts actor-critic for discrete action spaces with importance sampling corrections to account for the policy mismatch between when data was collected and when it's used.

  • SAC (Soft Actor-Critic): Designed for continuous actions, SAC maximizes both reward AND entropy (exploration bonus), making it exceptionally stable and sample-efficient.

Why This Matters

Off-policy methods can reuse each environment interaction multiple times, making them 10-100x more sample efficient than their on-policy counterparts. This is critical when environment interactions are expensive (robotics, real-world systems) or slow (complex simulations).

The trade-off? Increased algorithmic complexity due to corrections needed when learning from data generated by old policies.

Key Takeaway: Off-policy actor-critic methods like ACER and SAC combine the strengths of policy gradients with experience replay from value-based methods, achieving superior sample efficiency at the cost of implementation complexity.