Course contentsShow
← Machine Learning and Deep Learning
Lesson 1786 of 3,53838. Instruction Tuning and AlignmentFree lesson

Multi-Objective Reward Models

Modeling multiple attributes: helpfulness, harmlessness, honesty, and combining preferences into a single reward.

Multi-Objective Reward Models

What you'll learn: How to build reward models that simultaneously optimize for multiple alignment goals like helpfulness, harmlessness, and honesty.

The Multi-Objective Challenge

So far, we've trained reward models to predict a single scalar score representing overall preference. But human preferences aren't one-dimensional—you want your LLM to be helpful (answering questions thoroughly), harmless (avoiding toxic or dangerous content), and honest (admitting uncertainty rather than hallucinating). These objectives sometimes conflict: a maximally helpful response might include risky information, while a maximally harmless one might be too cautious.

Modeling Multiple Attributes

Instead of outputting one reward score, a multi-objective reward model produces separate scores for each attribute:

# Single-objective (previous approach)
reward = reward_model(prompt, response)  # scalar

# Multi-objective
scores = reward_model(prompt, response)  
# scores = {"helpfulness": 0.8, "harmlessness": 0.9, "honesty": 0.7}

The architecture typically adds multiple output heads to the reward model, each trained on preference data labeled for that specific attribute. You need human annotators to rate responses along each dimension separately.

Combining Into a Single Reward

For RL fine-tuning, you still need one reward signal. Common approaches:

  • Weighted sum: total_reward = w₁·helpfulness + w₂·harmlessness + w₃·honesty
  • Constrained optimization: Maximize helpfulness while keeping harmlessness above a threshold
  • Pareto optimization: Find responses that improve multiple objectives simultaneously

Think of it like cooking: you want flavor (helpfulness), nutrition (harmlessness), and authenticity (honesty). The weights let you decide whether you're making comfort food or health food.

Key Takeaway: Multi-objective reward models decompose alignment into separate, measurable attributes, allowing fine-grained control over LLM behavior through weighted combinations or constrained optimization.