Course contentsShow
← Machine Learning and Deep Learning
Lesson 2328 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

Debugging Continuous Control Agents

Common failure modes: exploding actions, collapsed exploration, critic overestimation, and diagnostic techniques.

Debugging Continuous Control Agents

What you'll learn: How to diagnose and fix the most common failure modes that plague continuous control agents like DDPG, TD3, and SAC.

Why Continuous Control Agents Fail Differently

Continuous control brings unique debugging challenges compared to discrete action spaces. Small numerical instabilities can cascade into catastrophic failures. Let's examine the four most common failure modes and how to spot them.

Common Failure Modes

1. Exploding Actions

Your agent outputs actions like [999.0, -500.0, 1e6] instead of normalized values. This happens when your policy network lacks proper output constraints (like tanh activation) or when gradient updates become too large.

Diagnostic: Log action magnitudes. If they exceed your environment's bounds or grow over time, you have explosion.

2. Collapsed Exploration

The agent repeats the same action regardless of state—it's "frozen." Common in DDPG when exploration noise decays too quickly or the policy network converges to a deterministic bad local minimum.

Diagnostic: Check action variance across episodes. Near-zero variance = collapsed exploration.

3. Critic Overestimation

The critic (value function) predicts unrealistically high Q-values, causing the actor to chase impossible returns. This is why TD3 uses twin critics and clipped double Q-learning.

Diagnostic: Plot Q-value estimates vs actual returns. Large positive bias indicates overestimation.

4. Numerical Instability

NaN or Inf values appear in losses or network parameters, often from dividing by tiny probabilities (in SAC's entropy term) or unbounded advantage estimates.

Diagnostic: Add assertion checks for torch.isnan() and torch.isinf() after each gradient update.

Essential Diagnostic Techniques

  • Log everything: action statistics, Q-values, losses, gradient norms
  • Visualize policy evolution: render episodes at regular intervals
  • Start simple: test on easy environments like Pendulum before complex robotics tasks
  • Sanity checks: can your agent solve with random actions? Does it overfit to one episode?

Key Takeaway: Debugging continuous control requires monitoring action distributions, Q-value estimates, and gradient magnitudes—catching issues before they cascade into complete training failure.