Course contentsShow
← AI Engineering
Lesson 75 of 1,8862. Working with Pre-trained ModelsFree lesson

Understanding Device Placement in PyTorch

Learn how to move models and tensors between CPU, GPU, and other accelerators using PyTorch's device API.

Understanding Device Placement in PyTorch

You'll learn how to control where your models and data live—CPU, GPU, or other accelerators—to optimize inference speed and memory usage.

The Core Idea

Think of devices like different workspaces in a factory. The CPU is a versatile general-purpose workspace, while the GPU is a specialized assembly line built for parallel tasks. Your model and data must be in the same workspace to work together, and choosing the right workspace dramatically affects performance.

In PyTorch, every tensor and model has a "device" location. By default, everything starts on the CPU. Moving to a GPU can speed up inference by 10-100x for neural networks, but requires explicit device management.

How Device Placement Works

PyTorch uses a simple device API. You specify a device object, then move your tensors and models there:

import torch

# Check if GPU is available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Move a model (like from Transformers)
model = model.to(device)

# Move input tensors
inputs = inputs.to(device)

Critical rule: Both model and inputs must be on the same device, or PyTorch will throw an error. You can't process CPU data with a GPU model.

You can also specify devices directly: torch.device("cuda:0") for the first GPU, torch.device("cpu") for CPU.

Why This Matters for Inference

As you learned in "CPU vs GPU Inference Trade-offs" and "Memory Management During Inference," device placement directly impacts your throughput and latency. Smart device management ensures you're using the right hardware for each workload.

Key Takeaway: Device placement in PyTorch determines where computation happens—always ensure your model and data are on the same device, and choose GPU when you need parallel processing speed.