Course contentsShow
← Machine Learning and Deep Learning
Lesson 825 of 3,53819. PyTorch FundamentalsFree lesson

Prefetching and DataLoader Performance Tuning

Optimize DataLoader with prefetch_factor, persistent_workers, and profiling to maximize throughput.

Prefetching and DataLoader Performance Tuning

What You'll Learn

You'll discover how to squeeze maximum throughput from PyTorch's DataLoader by configuring prefetching settings and identifying bottlenecks through profiling.

The Performance Challenge

Even with num_workers and pin_memory configured, your GPU might sit idle waiting for data. Prefetching solves this by preparing batches ahead of time—like a restaurant mise en place where ingredients are prepped before orders arrive.

Key Tuning Parameters

prefetch_factor

Controls how many batches each worker loads in advance. Default is 2, meaning each worker prepares 2 batches while the model trains on the current one.

dataloader = DataLoader(
    dataset,
    batch_size=32,
    num_workers=4,
    prefetch_factor=3,  # Each worker prefetches 3 batches
    pin_memory=True
)

Higher values increase memory usage but reduce waiting time. With 4 workers and prefetch_factor=3, you have 12 batches ready!

persistent_workers

Keeps worker processes alive between epochs instead of respawning them, eliminating startup overhead.

dataloader = DataLoader(
    dataset,
    batch_size=32,
    num_workers=4,
    persistent_workers=True  # Workers survive across epochs
)

Critical for small datasets with many epochs where worker initialization dominates time.

Profiling Your Pipeline

Use PyTorch's profiler to identify whether your GPU is data-starved:

from torch.profiler import profile, ProfilerActivity

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:
    for batch in dataloader:
        # training step
        pass

print(prof.key_averages().table())

Look for high CPU wait times—signals to increase prefetch_factor or num_workers.

Key Takeaway: Optimize DataLoader throughput by tuning prefetch_factor (memory-for-speed trade-off) and enabling persistent_workers (eliminate startup costs), then validate improvements with profiling to ensure your GPU stays fed.