Course contentsShow
← AI Engineering
Lesson 66 of 1,8862. Working with Pre-trained ModelsFree lesson

Torch Compile and JIT

Using PyTorch's compilation features to optimize model execution with torch.compile().

Torch Compile and JIT

What you'll learn: How to use PyTorch's built-in compilation features to make your models run faster without changing their behavior.

The Core Idea

When PyTorch runs your model normally, it executes operations one at a time, checking and interpreting each step as it goes. Think of it like reading a recipe line-by-line while cooking—functional, but not the fastest approach.

torch.compile() and JIT (Just-In-Time compilation) work like meal-prepping: they analyze your entire model ahead of time, optimize the execution path, remove redundant operations, and create a faster version. The model still does the same thing, but more efficiently.

How torch.compile() Works

Introduced in PyTorch 2.0, torch.compile() is the modern, simpler approach. You wrap your model once, and PyTorch automatically optimizes it:

import torch

# Your model (already loaded from previous lessons)
model = AutoModelForCausalLM.from_pretrained("model-name")

# Compile it for faster inference
compiled_model = torch.compile(model)

# Use it exactly the same way
outputs = compiled_model(input_ids)

The compiled version analyzes your model's computation graph and applies optimizations like operation fusion (combining multiple operations into one) and memory layout improvements.

When This Matters

Compilation is especially valuable when you're running the same model repeatedly during inference (which you almost always are). The first run might be slower as PyTorch compiles, but subsequent runs get significantly faster—often 30-50% speedier.

Key Takeaway: torch.compile() is a one-line optimization that speeds up inference by pre-analyzing and optimizing your model's execution path, with no changes to your model's behavior or accuracy.