In this lesson, you will learn how the master-worker relationship between the NameNode and DataNodes enables HDFS to maintain high data availability through block replication.
The Roles: NameNode and DataNode
At the heart of HDFS are two distinct types of nodes. The NameNode acts as the "librarian" of the cluster. It holds the metadata—the directory tree and the mapping of which files consist of which blocks, and where those blocks are stored. Crucially, the NameNode does not store the actual file data.
The DataNodes are the "workers." They are responsible for storing the actual chunks of your data, known as blocks (typically 128MB or 256MB in size). When a client wants to read or write a file, it talks to the NameNode to find out which DataNodes hold the specific blocks it needs, then interacts directly with those DataNodes to perform the I/O.
Ensuring Durability via Replication
Because hardware failures are common in large distributed systems, HDFS ensures data durability through block replication. When a file is uploaded, HDFS breaks it into blocks and copies each block across multiple DataNodes (the default replication factor is 3).
The NameNode monitors the health of DataNodes via "heartbeats." If a DataNode goes offline, the NameNode detects the missing blocks and automatically instructs other DataNodes to replicate those blocks to new locations, restoring the cluster to the desired replication factor.
Visualization of the Strategy
Imagine a file with one block (Block_A). With a replication factor of 3, the NameNode ensures the cluster holds three copies:
Client Uploads File: "Data.csv"
NameNode splits file into: [Block_A]
NameNode orchestrates placement:
- DataNode_01: Stores [Block_A]
- DataNode_02: Stores [Block_A]
- DataNode_03: Stores [Block_A]
If DataNode_01 crashes, the NameNode identifies the loss and triggers a re-replication of Block_A from DataNode_02 or 03 to DataNode_04, maintaining the integrity of your data without manual intervention.
Key Takeaway: The NameNode manages the cluster's metadata map, while DataNodes provide the physical storage, and the block replication strategy ensures that even if individual servers fail, your data remains intact and accessible.