Contextual Bandits
What you'll learn: How to make smarter decisions by considering situational context, bridging simple bandits and full reinforcement learning.
The Core Idea
In the multi-armed bandit problems you've learned, every decision happens in a vacuum—you pull an arm without knowing anything about the current situation. Contextual bandits add a crucial piece: state information (called "context") that helps you choose better actions.
Think of it like this: A regular bandit is choosing which ad to show a user without knowing anything about them. A contextual bandit sees the user's age, location, browsing history, and then picks the ad. Same action space, but now context-aware.
From Bandits to RL
Mathematically, at each step you observe a context (or state) s, choose an action a, and receive reward r. Unlike full MDPs, there's no state transition—the next context is independent of your action. You're solving many related bandit problems simultaneously, one for each possible context.
This bridges the gap between:
- Multi-armed bandits: No state, just action → reward
- Contextual bandits: State → action → reward (no state transitions)
- Full RL (MDPs): State → action → reward → new state (with transitions)
Why It Matters
Contextual bandits are perfect for:
- Personalized recommendations (context = user features)
- Clinical trials (context = patient health metrics)
- Dynamic pricing (context = market conditions)
You can still use epsilon-greedy, UCB, or other exploration strategies, but now your value estimates are conditioned on context: Q(s, a) instead of just Q(a).
Key Takeaway: Contextual bandits extend multi-armed bandits by incorporating state information before action selection, creating a stepping stone toward full reinforcement learning while avoiding the complexity of state transitions.