In this lesson, you will learn how to pinpoint the root cause of java.lang.OutOfMemoryError (OOM) by interpreting Spark heap logs and using heap dumps to distinguish between driver and executor failures.
When a Spark job crashes with an OOM error, the first step is to identify where the memory is failing. If the failure occurs on the Driver, it is typically due to collecting large datasets (e.g., calling .collect() on a massive RDD) or storing excessive metadata from thousands of partitions. If the failure occurs on an Executor, the job is likely struggling with data skew, oversized broadcast joins, or insufficient memory allocated to the execution/storage segments.
Analyzing the Logs
Before diving into memory dumps, examine your executor logs. A log entry like ExecutorLostFailure usually indicates that the YARN or Kubernetes resource manager killed the container because it exceeded its allocated memory limit. Look for the "GC overhead limit exceeded" message; this signals that the Java Garbage Collector is spending all its time trying to free up space because the heap is permanently saturated.
Capturing Heap Dumps
When logs aren't enough, you need a snapshot of the memory state at the moment of failure. You can configure Spark to automatically generate a heap dump when an OOM occurs by adding the following to your spark-submit or configuration file:
--conf "spark.executor.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/executor.hprof"
Once you have the .hprof file, use tools like Eclipse MAT (Memory Analyzer Tool). By opening the dump, you can run the "Leak Suspects" report. This will immediately show you if the memory is being held by a specific object, such as a massive ArrayList of objects being collected by the driver or an oversized hash table created during a sort-merge join on an executor.
If your heap dump shows a massive TaskMemoryManager allocation, you know your task-level operations are too memory-intensive. If it shows large object arrays held by the driver, you are likely pulling too much data out of the cluster.
Key Takeaway: Diagnose OOM errors by using automatic heap dumps to identify whether memory pressure is caused by excessive driver collection or executor-side data skew.