In this lesson, you will learn how the physical movement of data between storage and memory—either sequentially or randomly—dramatically impacts the throughput of your data pipelines.
The Physical Reality of I/O
At the hardware level, data retrieval isn't instantaneous. Whether you are using traditional spinning hard drives (HDDs) or modern Solid State Drives (SSDs), the storage controller must locate the data before it can be read.
Sequential I/O
Sequential I/O occurs when the system reads data stored in contiguous, neighboring blocks. Think of this like reading a printed book from cover to cover; your eyes move predictably across the lines without skipping around.
In data engineering, sequential I/O is the gold standard for high throughput. When you perform a full table scan or read a large log file, the disk head (or controller) anticipates the next piece of data, often pre-fetching blocks into a cache to keep the transfer speed high.
Random I/O
Random I/O occurs when the system must jump to disparate, non-contiguous locations to retrieve small pieces of data. This is akin to using a dictionary; you constantly jump to different pages to find individual words.
This is significantly slower because the storage device must "seek" to a new location for every request. If your pipeline is performing thousands of small, scattered lookups, the time spent "seeking" will dwarf the actual time spent reading the data.
Why it Matters
Consider a simple Python pseudocode example where we compare reading data:
# Sequential: Reading a large Parquet file (Columnar Storage)
data = read_large_file("fact_sales.parquet")
# Result: High throughput, few 'seek' operations.
# Random: Fetching 1,000 individual records by ID from a fragmented heap
for id in list_of_ids:
row = db.get_by_id(id)
# Result: High latency, constant 'seek' operations as the head jumps around.
When designing your pipelines, favor batch processing and columnar formats—like those discussed in previous lessons—because they naturally promote sequential access. Use random access sparingly, usually reserved for point lookups where indexes (like B-Trees) can minimize the number of jumps required to find the target data.
Key Takeaway: Sequential I/O maximizes throughput by reading contiguous data, whereas random I/O introduces latency through frequent, scattered access to non-adjacent storage locations.