Collate Functions: Custom Batch Creation
What You'll Learn
You'll learn how to write custom collate functions that transform lists of dataset samples into properly structured batches, handling tricky cases like variable-length sequences and complex data formats.
The Problem DataLoader Solves (And Creates)
By default, PyTorch's DataLoader stacks your samples into batches using a simple default collate function. This works perfectly when all samples have identical shapes—but real-world data often doesn't cooperate. Text sequences have different lengths, images might vary in size, or you might have complex nested structures that need special handling.
What Is a Collate Function?
A collate function is a custom function you pass to DataLoader that controls exactly how individual samples get combined into a batch. It receives a list of samples (whatever your __getitem__ returns) and must return a single batch structure.
Think of it like organizing groceries: the default method just stacks boxes uniformly, but a custom collate function knows to carefully arrange fragile items, group frozen goods, and handle oddly-shaped packages.
Common Use Cases
Variable-length sequences: Pad text or time-series data to the same length within each batch, creating a tensor plus a mask indicating real vs padded values.
Complex structures: When samples contain multiple elements (image, caption, metadata), collate functions organize them into separate batch tensors or dictionaries.
Dynamic padding: Instead of padding all sequences to a global maximum, pad only to the longest sequence in that specific batch, saving memory and computation.
Implementation Pattern
def custom_collate_fn(batch):
# batch is a list of samples from __getitem__
# Return properly structured batch tensors
pass
loader = DataLoader(dataset, batch_size=32, collate_fn=custom_collate_fn)
Key Takeaway: Custom collate functions give you precise control over batch creation, letting you handle variable-length data, apply padding strategies, and structure complex batches exactly as your model requires.