One-Step Actor-Critic Algorithm
What you'll learn: How to implement the simplest actor-critic method that updates both the policy and value function after every single action using TD(0).
The Core Idea
The One-Step Actor-Critic algorithm is the most straightforward combination of actor (policy) and critic (value function). After each action, you immediately update both networks using the TD error from that single transition. Think of it like a student (actor) who performs an action and instantly gets feedback from a teacher (critic) about whether that action was better or worse than expected.
How It Works
- Take one action using your current policy (actor)
- Observe the reward and next state
- Compute the TD error: δ = r + γV(s') - V(s)
- Update the critic to make V(s) closer to the bootstrapped target r + γV(s')
- Update the actor using the policy gradient scaled by δ (the advantage estimate)
The TD error δ serves double duty: it tells the critic how wrong its value estimate was, and it tells the actor whether the action was better (positive δ) or worse (negative δ) than expected.
Why "One-Step"?
Unlike REINFORCE which waits for entire episodes, this algorithm uses TD(0): only the immediate reward plus the estimated value of the next state. It's faster and more responsive but relies heavily on the critic's current (possibly inaccurate) value estimates.
This is an on-policy method—you learn from actions your current policy takes. Each transition provides a learning signal, making it sample-efficient for online learning scenarios.
Key Takeaway: The One-Step Actor-Critic updates both policy and value function after every action using TD(0), providing immediate feedback but depending on bootstrapped value estimates for advantage computation.