Continuous Action Spaces
What you'll learn: How to handle reinforcement learning problems where actions aren't discrete choices but continuous values, like steering angles or motor torques.
The Problem with Discrete Actions
So far, most RL algorithms you've seen work with discrete action spaces—choosing from a finite set of actions (up, down, left, right). But many real-world tasks require continuous actions: a robot arm's joint angle can be any value between -180° and +180°, a self-driving car's steering wheel can turn to any degree, or a drone's throttle can vary smoothly from 0% to 100%.
Gaussian Parameterization: The Standard Solution
Instead of outputting separate probabilities for each discrete action, we parameterize the policy to output the parameters of a continuous distribution—typically a Gaussian (normal) distribution.
Your policy network now outputs:
- Mean (μ): The "center" of your action distribution
- Standard deviation (σ): How much exploration/randomness around that mean
When your agent needs to act, it samples from this Gaussian distribution: action ~ N(μ, σ²).
Why This Works
Think of it like a basketball player taking free throws. The mean μ is where they're aiming, and σ is how consistent they are. Early in training, σ is high (lots of exploration, varying shots). As they improve, σ decreases (more precise, confident actions), while μ shifts toward the optimal target.
The policy gradient still applies! You differentiate through the log-probability of the Gaussian to update both μ and σ based on rewards received.
Key Takeaway: Continuous action spaces are handled by having your policy output Gaussian distribution parameters (mean and standard deviation), then sampling actions from that distribution—enabling smooth control in tasks like robotics and continuous control.