Course contentsShow
← Machine Learning and Deep Learning
Lesson 824 of 3,53819. PyTorch FundamentalsFree lesson

Iterable Datasets for Streaming Data

Use IterableDataset for large-scale or streaming data that doesn't fit in memory or lacks random access.

Iterable Datasets for Streaming Data

You'll learn when and how to use PyTorch's IterableDataset for data that can't be loaded all at once or accessed randomly.

The Core Problem

So far, you've worked with standard Dataset classes that assume all your data either fits in memory or can be randomly accessed by index (like dataset[42]). But what if you're working with:

  • Massive datasets that exceed available RAM
  • Data streams from sensors, logs, or APIs
  • Files that must be read sequentially (like large text files)
  • Database query results that arrive one chunk at a time

This is where IterableDataset shines.

What Makes It Different

Think of a regular Dataset like a bookshelf where you can grab any book by its position. An IterableDataset is more like a conveyor belt—items flow past you one by one, and you can't easily jump backwards or forwards.

IterableDataset provides data through iteration (a __iter__ method) rather than indexing (__getitem__). You loop through it once, processing items as they arrive:

from torch.utils.data import IterableDataset

class StreamingDataset(IterableDataset):
    def __init__(self, data_stream):
        self.data_stream = data_stream
    
    def __iter__(self):
        for item in self.data_stream:
            # Process and yield data on-the-fly
            yield process(item)

You use it with DataLoader just like regular datasets, but shuffling and random access aren't available—the data flows in order.

When to Use It

Choose IterableDataset when your data source is naturally sequential, too large for memory, or arrives continuously. It's perfect for log analysis, sensor streams, or processing terabytes of data that would never fit into RAM.

Key Takeaway: IterableDataset enables streaming workflows where data flows through your pipeline sequentially, trading random access for the ability to handle unlimited data volumes.