SAC: Algorithm and Architecture
What you'll learn: How Soft Actor-Critic implements maximum entropy RL through its unique combination of stochastic policies, twin critics, and automatic temperature tuning.
Core Components
You've learned that SAC embraces the maximum entropy framework. Now let's see how it's actually built.
Stochastic Actor with Reparameterization Trick
Unlike DDPG's deterministic policy, SAC's actor outputs a Gaussian distribution over actions. Instead of sampling directly (which breaks gradients), SAC uses the reparameterization trick: it outputs mean μ and std σ, then computes action = μ + σ * ε where ε ~ N(0,1). This separates randomness from learnable parameters, allowing gradients to flow through the sampling operation.
Analogy: Rather than rolling a biased die (non-differentiable), you roll a fair die and then apply a learnable transformation to bias it (differentiable).
Twin Q-Critics
SAC borrows TD3's twin critic networks to combat overestimation bias. Two Q-functions are trained independently, and the minimum of their predictions is used when computing targets and policy updates. This pessimistic estimate reduces false optimism.
Automatic Temperature Tuning
The temperature parameter α controls the entropy-reward trade-off. Instead of manually tuning it, SAC treats α as a learnable parameter optimized to match a target entropy (typically -dim(action_space)). As training progresses, α adjusts automatically to maintain desired exploration levels.
Architecture Flow
- Actor samples actions using the reparameterization trick
- Both critics evaluate state-action pairs
- Actor maximizes
Q_min(s,a) - α*log π(a|s)(value minus entropy cost) - Critics minimize TD error using target networks
- Temperature
αadjusts to maintain target entropy
Key Takeaway: SAC combines a stochastic reparameterized actor, twin Q-critics for stability, and automatic temperature adjustment to achieve robust, sample-efficient learning in continuous control without manual hyperparameter tuning of the exploration-exploitation balance.