Iterable Datasets for Streaming Data
You'll learn when and how to use PyTorch's IterableDataset for data that can't be loaded all at once or accessed randomly.
The Core Problem
So far, you've worked with standard Dataset classes that assume all your data either fits in memory or can be randomly accessed by index (like dataset[42]). But what if you're working with:
- Massive datasets that exceed available RAM
- Data streams from sensors, logs, or APIs
- Files that must be read sequentially (like large text files)
- Database query results that arrive one chunk at a time
This is where IterableDataset shines.
What Makes It Different
Think of a regular Dataset like a bookshelf where you can grab any book by its position. An IterableDataset is more like a conveyor belt—items flow past you one by one, and you can't easily jump backwards or forwards.
IterableDataset provides data through iteration (a __iter__ method) rather than indexing (__getitem__). You loop through it once, processing items as they arrive:
from torch.utils.data import IterableDataset
class StreamingDataset(IterableDataset):
def __init__(self, data_stream):
self.data_stream = data_stream
def __iter__(self):
for item in self.data_stream:
# Process and yield data on-the-fly
yield process(item)
You use it with DataLoader just like regular datasets, but shuffling and random access aren't available—the data flows in order.
When to Use It
Choose IterableDataset when your data source is naturally sequential, too large for memory, or arrives continuously. It's perfect for log analysis, sensor streams, or processing terabytes of data that would never fit into RAM.
Key Takeaway: IterableDataset enables streaming workflows where data flows through your pipeline sequentially, trading random access for the ability to handle unlimited data volumes.