In this lesson, you will learn how to align your Spark executor resources with your physical hardware to maximize task parallelism while preventing memory contention.
The Balancing Act
When configuring Spark, the goal is to optimize the "Goldilocks zone" for your executors. If you assign too few cores, you underutilize your cluster; if you assign too many, you risk contention for memory, CPU cache, and garbage collection (GC) overhead.
The core relationship to balance is the ratio of spark.executor.cores to spark.executor.memory. A common pitfall is allocating too many cores per executor (e.g., > 5). Because Spark tasks share the same JVM heap within an executor, having too many concurrent tasks can lead to excessive GC pressure, as the available heap is split across more threads.
Practical Configuration Strategy
For most production workloads, we aim for 4 to 5 cores per executor. This allows for enough parallelism to keep the CPU busy while maintaining sufficient heap space for each task’s shuffle data and cache requirements.
Consider a node with 16 cores and 64GB of RAM. Instead of creating one giant executor, you might configure four executors per node:
# Example configuration for a balanced node
spark.executor.cores=4
spark.executor.memory=14g # Leave overhead for OS and off-heap memory
spark.executor.instances=...
By capping your cores at 4, you ensure that the Execution Memory—used for sorts and joins—is not fragmented by too many competing threads. If you find your jobs are failing with OutOfMemoryError even after increasing memory, don't just add more RAM; try reducing the number of cores per executor to lower the memory contention within the JVM.
Hardware Awareness
Always subtract the YARN or Kubernetes overhead (the "container overhead") from your total node memory. Your spark.executor.memory setting should generally be about 10-15% lower than the actual container size to ensure your executors don't get killed by the cluster manager for exceeding their memory limits.
Key Takeaway: Balance your executor cores at 4–5 to minimize JVM contention, and always leave a memory buffer for container overhead to ensure stability under load.