Course contentsShow
← Machine Learning and Deep Learning
Lesson 2139 of 3,53846. Reinforcement Learning: FundamentalsFree lesson

Episodes vs Continuing Tasks

Distinguishing finite-horizon episodic tasks from infinite-horizon continuing tasks.

Episodes vs Continuing Tasks

What you'll learn: How reinforcement learning problems differ based on whether they have a defined ending or run indefinitely.

The Core Distinction

In reinforcement learning, tasks fall into two fundamental categories based on their temporal structure:

Episodic tasks have a clear beginning and end. The agent interacts with the environment until it reaches a terminal state, at which point the episode concludes and everything resets. Think of a game of chess: the episode starts with the initial board position and ends when someone wins or draws. Each game is self-contained.

Continuing tasks have no natural endpoint—they run indefinitely. The agent must optimize its behavior over an infinite time horizon. Imagine a thermostat controlling your home's temperature: it never "finishes" its job; it continuously makes decisions to maintain comfort.

Why This Matters

This distinction affects how we define returns and value functions:

  • Episodic tasks can use undiscounted returns (γ=1) or discounted returns, since the finite horizon prevents infinite accumulation
  • Continuing tasks typically require a discount factor γ < 1 to ensure returns remain bounded over infinite time steps

The episode structure also influences learning:

  • Episodic agents can learn from complete trajectories, resetting after each episode
  • Continuing agents must learn online, without natural "checkpoints" for evaluation

The Gray Area

Some tasks can be framed either way. A robot navigating to a goal could be episodic (episode ends at goal) or continuing (robot keeps operating after reaching goal). Your problem formulation depends on your objectives.

Key Takeaway: Episodic tasks have terminal states and natural resets, while continuing tasks run indefinitely—this distinction fundamentally shapes how we compute returns, apply discount factors, and design learning algorithms.