In this lesson, you will learn how to select the right tool for adjusting your data’s parallelism to optimize cluster performance.
When working with distributed datasets in Spark, you often need to adjust the number of partitions to match your workload. We do this using coalesce() and repartition(). While both change the number of partitions, they do so with very different performance implications.
The Efficiency of Coalesce
coalesce() is specifically designed for reducing the number of partitions. Because it avoids a full shuffle whenever possible, it is much faster than repartition(). It works by merging existing partitions on their current executor nodes.
Use coalesce() when:
- You want to decrease the number of partitions (e.g., before writing output files).
- You want to minimize network I/O and CPU overhead.
# Efficiently reduce to 5 partitions without a full shuffle
df_reduced = df.coalesce(5)
The Versatility of Repartition
repartition() is a "heavy" operation. It triggers a full shuffle across the cluster, meaning data is redistributed across all available nodes. While expensive, this is necessary when you need to change the data distribution entirely.
Use repartition() when:
- You need to increase the number of partitions to gain more parallelism.
- Your data is skewed, and you need to rebalance it evenly across the cluster.
- You are filtering a large dataset into a much smaller one and need to redistribute the remaining rows.
# Full shuffle to increase parallelism or rebalance data
df_rebalanced = df.repartition(100)
Choosing the Right Path
Think of coalesce() as a "local" operation—it’s like moving items from several nearby boxes into one. Think of repartition() as a "global" operation—it’s like emptying all the boxes into a common pile and then sorting them back into new boxes. Because repartition() involves moving data across the network, always prefer coalesce() if you only need to reduce the partition count. If you need to spread your data thin to prevent bottlenecking, pay the price of the shuffle with repartition().
Key Takeaway: Use coalesce() to reduce partitions efficiently by avoiding shuffles, and reserve repartition() for when you must increase parallelism or rebalance data across the cluster.