In this lesson, you will learn how the Spark DAG Scheduler translates your optimized logical plan into a sequence of execution stages and granular tasks.
When you trigger an action on a DataFrame, the Spark Driver’s DAG Scheduler steps in to convert the high-level plan into a physical execution model. The scheduler analyzes the Directed Acyclic Graph (DAG) to identify where data movement (shuffling) is required and where it is not.
Decomposing into Stages
The scheduler divides the DAG into Stages based on wide transformations. A stage boundary is created whenever a shuffle is required—such as during a groupBy or a join.
- Narrow Transformations: (e.g.,
filter,select,map) allow operations to stay within the same partition, meaning these can be pipelined together into a single stage. - Wide Transformations: (e.g.,
repartition,sort,groupBy) force a barrier where the scheduler must break the chain, as the output of one stage must be fully computed and exchanged across the cluster before the next stage can begin.
Task Distribution
Once the stages are defined, the scheduler decomposes each stage into individual Tasks. A task is the smallest unit of work, representing the execution of a specific computation on a single partition of data.
Consider this snippet:
# A single stage containing narrow transformations
df.filter(col("amount") > 100).select("id", "amount")
# A shuffle occurs here, creating a new stage boundary
.groupBy("id").sum("amount")
The scheduler sees the groupBy as a trigger for a new stage. All operations before the groupBy are packed into Stage 1, while the aggregation is assigned to Stage 2. The DAG Scheduler then maps these tasks to available executors based on data locality. If the data already exists on a specific executor, the scheduler prefers to schedule the task there to minimize network I/O, effectively turning your abstract logic into a highly parallelized workflow.
Key Takeaway: The DAG Scheduler breaks logical plans into stages at shuffle boundaries, then distributes the resulting tasks to executors to ensure parallel execution across the cluster.