Course contentsShow
← Machine Learning and Deep Learning
Lesson 2323 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

SAC: Algorithm and Architecture

SAC's stochastic actor with reparameterization trick, twin Q-critics, and automatic temperature tuning.

SAC: Algorithm and Architecture

What you'll learn: How Soft Actor-Critic implements maximum entropy RL through its unique combination of stochastic policies, twin critics, and automatic temperature tuning.

Core Components

You've learned that SAC embraces the maximum entropy framework. Now let's see how it's actually built.

Stochastic Actor with Reparameterization Trick

Unlike DDPG's deterministic policy, SAC's actor outputs a Gaussian distribution over actions. Instead of sampling directly (which breaks gradients), SAC uses the reparameterization trick: it outputs mean μ and std σ, then computes action = μ + σ * ε where ε ~ N(0,1). This separates randomness from learnable parameters, allowing gradients to flow through the sampling operation.

Analogy: Rather than rolling a biased die (non-differentiable), you roll a fair die and then apply a learnable transformation to bias it (differentiable).

Twin Q-Critics

SAC borrows TD3's twin critic networks to combat overestimation bias. Two Q-functions are trained independently, and the minimum of their predictions is used when computing targets and policy updates. This pessimistic estimate reduces false optimism.

Automatic Temperature Tuning

The temperature parameter α controls the entropy-reward trade-off. Instead of manually tuning it, SAC treats α as a learnable parameter optimized to match a target entropy (typically -dim(action_space)). As training progresses, α adjusts automatically to maintain desired exploration levels.

Architecture Flow

  1. Actor samples actions using the reparameterization trick
  2. Both critics evaluate state-action pairs
  3. Actor maximizes Q_min(s,a) - α*log π(a|s) (value minus entropy cost)
  4. Critics minimize TD error using target networks
  5. Temperature α adjusts to maintain target entropy

Key Takeaway: SAC combines a stochastic reparameterized actor, twin Q-critics for stability, and automatic temperature adjustment to achieve robust, sample-efficient learning in continuous control without manual hyperparameter tuning of the exploration-exploitation balance.