Course contentsShow
← Machine Learning and Deep Learning
Lesson 1236 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

Further Fine-Tuning: Starting from Base or Instruction

Deciding whether to fine-tune from base or instruction-tuned checkpoints for custom applications.

Further Fine-Tuning: Starting from Base or Instruction

What You'll Learn

You'll learn how to decide whether to start your custom fine-tuning from a base model or an instruction-tuned checkpoint.

The Core Decision

When building a custom application, you face an important choice: should you fine-tune from a base model (like GPT-3 base) or from an instruction-tuned model (like GPT-3.5-turbo or ChatGPT)?

Think of it like choosing between two different starting points for a road trip:

  • Base model: Starting from a raw highway—you have maximum freedom to go anywhere, but you need to do more navigation work
  • Instruction-tuned model: Starting from a guided tour route—easier to follow and modify, but some routes are already preferred

When to Start from Base

Choose a base model when:

  • You have large amounts of domain-specific data to fine-tune on
  • Your task involves completion or generation without instruction-following
  • You want maximum flexibility and don't need instruction-following behavior
  • You're doing continued pretraining for domain adaptation before task-specific fine-tuning

When to Start from Instruction-Tuned

Choose an instruction-tuned checkpoint when:

  • You have limited training data (few-shot scenarios)
  • Your task naturally fits an instruction-response format
  • You want the model to maintain general helpfulness while learning your specific task
  • You're building conversational or assistant-like applications

The Practical Reality

Starting from instruction-tuned models often requires less data because they've already learned to follow instructions. However, they may be more resistant to learning behaviors that conflict with their safety training or instruction-following patterns.

Key Takeaway: Start from base models when you have substantial domain data and want maximum flexibility; start from instruction-tuned models when you want to preserve helpful, instruction-following behavior with less training data.