Continuous Action Spaces: Fundamentals
What you'll learn: How continuous action spaces differ from discrete ones and why they require specialized policy gradient approaches.
Discrete vs Continuous Actions
In discrete action spaces (like Atari games), an agent chooses from a finite set of actions: move left, move right, jump, etc. You've already worked with these using methods like DQN and basic policy gradients.
Continuous action spaces are fundamentally different. Here, actions are real-valued vectors drawn from a continuous range. Think of a robotic arm: instead of "move left" or "move right," you specify exact joint angles like [23.7°, -15.2°, 40.1°]. Or consider self-driving cars controlling steering angle (any value between -30° and +30°) and throttle (0.0 to 1.0).
Why This Changes Everything
In discrete spaces, your policy network outputs a probability distribution over a finite set of actions (using softmax). You can easily compute probabilities for each action.
In continuous spaces, there are infinitely many possible actions. You can't output a probability for every possible steering angle—there are uncountably many! Instead, your policy must output the parameters of a probability distribution (like mean and standard deviation of a Gaussian), then sample from it.
Representation Challenges
This changes how we represent and compute with policies:
- Action selection: Sample from a continuous distribution rather than argmax or categorical sampling
- Probability computation: Evaluate probability density (not discrete probability) for gradient calculations
- Exploration: Controlled via distribution variance, not epsilon-greedy
Key Takeaway: Continuous action spaces require policies to output distribution parameters rather than action probabilities, fundamentally changing how we sample actions and compute policy gradients—adaptations you'll explore in specialized algorithms designed for this setting.