Course contentsShow
← Data Science
Lesson 1772 of 2,14542. Working with Big DataFree lesson

Data Locality and Network Bottlenecks

Moving computation to data rather than moving data, minimizing network transfer overhead.

Data Locality and Network Bottlenecks

What you'll learn: How bringing computation to your data—rather than shuttling data across networks—dramatically improves performance in distributed systems.

The Core Problem

In distributed big data systems, you're working with data spread across many machines. The naïve approach might be to gather all the data onto one node for processing, but this creates a massive network bottleneck. Moving terabytes of data across a network is slow—often the slowest part of your entire pipeline.

Data Locality: The Golden Rule

Data locality means running your computation where the data already lives. Instead of fetching data from 20 machines to process it in one place, you send lightweight processing instructions to each of those 20 machines and let them compute locally. Then you only transfer the much smaller results back.

Why This Matters

Think of it like a library analogy: Instead of checking out 1,000 books to read at home, you hire 10 readers to visit the library and each summarize 100 books on-site. You only need to collect their short summaries, not move mountains of books.

Network bandwidth is precious. A single modern hard drive can read data at ~200 MB/s, but network throughput between nodes might be only 10–100 MB/s when shared among many operations. Moving computation is cheap; moving data is expensive.

In the MapReduce model and shared-nothing architectures you've learned, schedulers actively try to assign tasks to nodes that already hold the relevant data partitions, minimizing cross-network transfers.

Key Takeaway: Maximize data locality by executing computations on the same nodes where data resides—this avoids network bottlenecks and keeps your distributed system fast and efficient.