Reward Functions and Signals
What You'll Learn
You'll learn how scalar reward values guide reinforcement learning agents toward desired behaviors by signaling which actions lead to successful outcomes.
The Core Idea
In a Markov Decision Process, the reward function R(s, a, s') produces a scalar (single number) signal that tells the agent how "good" or "bad" a particular transition was. When the agent takes action a in state s and arrives at new state s', it receives a numerical reward.
Think of rewards like feedback scores in a video game: +10 points for collecting a coin, -5 for taking damage, +100 for completing a level. The agent doesn't know the rules of the game explicitly—it only sees these numbers and learns to maximize them over time.
Designing Reward Functions
The reward function is how you, the system designer, communicate goals to your agent:
- Sparse rewards: Only non-zero at goal states (e.g., +1 for winning, 0 otherwise)
- Dense rewards: Frequent feedback at many steps (e.g., distance-to-goal improvements)
- Shaped rewards: Carefully crafted intermediate rewards to guide learning
The Reward Hypothesis
Reinforcement learning operates on the principle that goals can be expressed as maximizing cumulative reward. Whether you're training a robot to walk, an agent to play chess, or a chatbot to satisfy users, you must translate that objective into a reward signal.
Common Pitfalls
Poor reward design leads to reward hacking—the agent finds unintended shortcuts that maximize reward without achieving your true goal. For example, a cleaning robot might learn to cover its dirt sensor instead of actually cleaning.
Key Takeaway: The reward function R(s, a, s') is the scalar signal that encodes your objectives, shaping agent behavior by providing numerical feedback for each state transition—designing it well is crucial to learning the right behavior.