Course contentsShow
← Machine Learning and Deep Learning
Lesson 2997 of 3,53865. LLM Inference EnginesFree lesson

Creating Draft Models: Distillation Approaches

Training smaller draft models through knowledge distillation from target models to maximize acceptance rates for specific model families.

Creating Draft Models: Distillation Approaches

What you'll learn: How to train specialized, lightweight draft models using knowledge distillation from your target LLM to maximize speculative decoding acceptance rates.

Why Distillation for Draft Models?

You've learned that speculative decoding requires a fast draft model to propose tokens. But where do these draft models come from? The best approach is knowledge distillation: training a smaller student model to mimic your specific target model's behavior.

The Core Idea

Think of it like training an apprentice chef (draft model) to predict exactly what a master chef (target model) would cook, not just to cook well in general. The goal isn't generic language modeling performance—it's maximum alignment with your target model's distribution to boost acceptance rates.

How It Works

  1. Use target model as teacher: Run your target LLM on training data to generate probability distributions over tokens
  2. Train student to match distributions: The draft model learns to mimic these distributions using techniques like:
    • Output distribution matching: Minimize KL-divergence between draft and target logits
    • Sequence-level distillation: Train on target model's actual generated sequences
  3. Architecture selection: Create a smaller, faster architecture (fewer layers, smaller hidden dimensions) from the same model family
  4. Shared vocabulary: Ensure the draft model uses identical tokenization for seamless integration

Why This Maximizes Acceptance

A distilled draft model learns your target's specific quirks and preferences—not just general language patterns. When the draft proposes tokens, they're statistically closer to what the target would have chosen, leading to higher acceptance rates and better speedup.

Key Takeaway: Knowledge distillation creates draft models specifically tuned to your target model's distribution, maximizing speculative decoding acceptance rates far beyond generic smaller models from the same family.