Episodes vs Continuing Tasks
What you'll learn: How reinforcement learning problems differ based on whether they have a defined ending or run indefinitely.
The Core Distinction
In reinforcement learning, tasks fall into two fundamental categories based on their temporal structure:
Episodic tasks have a clear beginning and end. The agent interacts with the environment until it reaches a terminal state, at which point the episode concludes and everything resets. Think of a game of chess: the episode starts with the initial board position and ends when someone wins or draws. Each game is self-contained.
Continuing tasks have no natural endpoint—they run indefinitely. The agent must optimize its behavior over an infinite time horizon. Imagine a thermostat controlling your home's temperature: it never "finishes" its job; it continuously makes decisions to maintain comfort.
Why This Matters
This distinction affects how we define returns and value functions:
- Episodic tasks can use undiscounted returns (γ=1) or discounted returns, since the finite horizon prevents infinite accumulation
- Continuing tasks typically require a discount factor γ < 1 to ensure returns remain bounded over infinite time steps
The episode structure also influences learning:
- Episodic agents can learn from complete trajectories, resetting after each episode
- Continuing agents must learn online, without natural "checkpoints" for evaluation
The Gray Area
Some tasks can be framed either way. A robot navigating to a goal could be episodic (episode ends at goal) or continuing (robot keeps operating after reaching goal). Your problem formulation depends on your objectives.
Key Takeaway: Episodic tasks have terminal states and natural resets, while continuing tasks run indefinitely—this distinction fundamentally shapes how we compute returns, apply discount factors, and design learning algorithms.