Course contentsShow
← Machine Learning and Deep Learning
Lesson 2146 of 3,53846. Reinforcement Learning: FundamentalsFree lesson

Formulating Real Problems as MDPs

Practical considerations: choosing state representations, action spaces, and reward shaping.

Formulating Real Problems as MDPs

What you'll learn: How to translate real-world problems into the formal MDP framework by carefully choosing states, actions, and rewards.

The Art of MDP Design

You've learned what MDPs are in theory—now comes the creative part: turning messy real problems into clean MDP formulations. This is more art than science, requiring thoughtful choices about what information to track, what actions the agent can take, and how to shape rewards to guide learning.

Three Critical Design Decisions

State Representation

Choose what information the agent observes. Too little, and you violate the Markov property (future depends on unobserved history). Too much, and you face the "curse of dimensionality"—enormous state spaces that are impossible to learn.

Example: For a robot vacuum, you might include current position, battery level, and dirt locations—but probably not the exact time of day or wall color.

Action Space

Define what the agent can do. Actions should be:

  • Executable: The agent can actually perform them
  • Granular enough: To enable precise control
  • Not too fine-grained: Avoid making simple tasks require hundreds of steps

Example: A trading agent might have actions like "buy," "sell," "hold" rather than specifying exact dollar amounts.

Reward Shaping

Design rewards that incentivize desired behavior without unintended consequences. Sparse rewards (only at goal) are simple but hard to learn from. Dense rewards (every step) guide learning but risk reward hacking—the agent exploits loopholes instead of solving your actual problem.

Example: For a game-playing agent, reward +1 for winning is sparse. Adding +0.01 per enemy defeated shapes behavior but might encourage farming weak enemies instead of progressing.

Key Takeaway: Formulating real problems as MDPs requires balancing expressiveness (capturing relevant information) with tractability (keeping state/action spaces manageable) while carefully shaping rewards to align learning with your true objective.