PyTorch Datasets: Creating Custom Dataset Classes
What You'll Learn
You'll learn how to create custom Dataset classes in PyTorch by implementing two essential methods that give your models flexible, standardized access to any data source.
The Core Idea
Think of a Dataset as a library catalog system. Just as a librarian needs to know (1) how many books exist and (2) how to retrieve any specific book by its catalog number, PyTorch needs your Dataset to answer two questions: "How many samples do you have?" and "Give me sample number X."
In PyTorch, you create custom Dataset classes by inheriting from torch.utils.data.Dataset and implementing two required methods:
__len__(self)- Returns the total number of samples in your dataset__getitem__(self, idx)- Returns the sample at indexidx
This simple interface allows PyTorch's DataLoader to efficiently batch, shuffle, and load your data during training, regardless of whether your data comes from images, text files, databases, or APIs.
The Basic Pattern
from torch.utils.data import Dataset
class MyCustomDataset(Dataset):
def __init__(self, data, labels):
# Store your data references
self.data = data
self.labels = labels
def __len__(self):
# Return total number of samples
return len(self.data)
def __getitem__(self, idx):
# Return one sample (input, target) pair
sample = self.data[idx]
label = self.labels[idx]
return sample, label
When you index into your dataset (dataset[5]), Python automatically calls __getitem__(5). The __len__ method enables len(dataset) to work properly.
Key Takeaway: Custom Dataset classes require only __len__() to report dataset size and __getitem__(idx) to retrieve individual samples, giving you complete control over data loading while maintaining compatibility with PyTorch's training infrastructure.