Computing Loss for Fine-Tuning Objectives
What you'll learn: How to calculate the loss function that guides your LLM during fine-tuning, whether for next-token prediction or task-specific objectives.
The Core Idea
During fine-tuning, your model needs feedback on how well it's performing. The loss function quantifies the difference between what your model predicts and what you want it to predict. This numerical signal tells the optimizer which direction to adjust parameters.
For LLMs, you typically compute loss in one of two ways:
Next-Token Prediction Loss
This is the same objective used in pretraining. Given a sequence from your fine-tuning dataset, you:
- Feed tokens through the forward pass
- For each position, compare the model's predicted probability distribution over vocabulary against the actual next token
- Use cross-entropy loss to measure how "wrong" the prediction was
Think of it like a teacher grading an essay question where the student had to fill in blanks—you penalize confidently wrong answers more than uncertain ones.
Task-Specific Loss Functions
For specialized tasks, you might compute different losses:
- Classification tasks: Cross-entropy over class labels (e.g., sentiment categories)
- Instruction following: Loss only on the model's response portion, ignoring the instruction tokens
- Preference learning: Ranking loss comparing preferred vs rejected outputs
The key insight: you're backpropagating through the exact same forward pass you just learned about, but the loss tells the model what to optimize for in your specific use case.
Why It Matters
The loss computation directly shapes what your model learns. Computing loss only on response tokens (masking instruction tokens) teaches the model to generate good answers without "memorizing" prompts unnecessarily.
Key Takeaway: The loss function transforms your fine-tuning objective into a numerical signal that drives parameter updates—typically next-token cross-entropy, but customizable for task-specific goals.