In this lesson, you will learn how to fine-tune the Spark Unified Memory Manager to balance the tug-of-war between cached data and the temporary buffers needed for shuffling and joins.
The Memory Tug-of-War
When you run a Spark application, the executor’s memory is divided into two primary regions: Storage and Execution.
- Storage Memory: Used for caching DataFrames (via
.cache()or.persist()) and storing broadcast variables. - Execution Memory: Used for buffers required during shuffles, joins, sorts, and aggregations.
In Spark’s Unified Memory Manager model, these two regions are not strictly partitioned by hard boundaries. Instead, they share a flexible pool. If Execution memory is empty, Storage can borrow space from it, and vice versa. However, when one region requires more memory while the other is full, the manager evicts cached blocks to free up space for execution—which can lead to expensive disk I/O if your cached data is dropped and needs to be recomputed later.
Tuning the Balance
You control this dynamic boundary using the spark.memory.fraction and spark.memory.storageFraction properties.
spark.memory.fraction(default 0.6): This is the percentage of the JVM heap used for the unified pool. The remaining 40% is reserved for internal metadata, user data structures, and preventing OutOfMemory errors.spark.memory.storageFraction(default 0.5): This sets the "soft" barrier. It determines the fraction of the unified pool that is reserved specifically for storage.
If your workload involves massive joins or shuffles, you might notice frequent "spilling to disk," which signifies that the execution region is starved. You can lower the storageFraction to allow execution tasks to claim more of that pool:
# Allow more room for shuffle/join buffers
--conf spark.memory.storageFraction=0.2
By reducing the storage fraction, you make it easier for execution tasks to "borrow" space from the storage region, preventing performance-killing disk spills at the cost of making your cached data more volatile. Always monitor the Spark UI "Executors" tab to see if your tasks are spilling—if they are, your memory configuration is likely your primary bottleneck.
Key Takeaway: The Unified Memory Manager dynamically balances space between storage and execution, but you can influence its bias by tuning spark.memory.storageFraction to prioritize high-throughput shuffles over persistent caching.