Addressing Data Skew in Distributed Joins

Learners will detect skewed data distributions in execution plans and apply techniques like salting to balance the workload across cluster nodes.

In this lesson, you will learn how to identify data skew in your execution plans and neutralize its performance impact using the "salting" technique.

The Bottleneck of Uneven Distribution

In a distributed join, your cluster splits data across nodes based on a join key. When one key (e.g., a NULL value or a "power user" ID) appears significantly more often than others, the node responsible for that key receives a disproportionately large slice of data.

While other nodes finish their tasks quickly, the "skewed" node struggles with an outsized memory load, often triggering spilling to disk or task failure due to OutOfMemory errors. In your execution plan, look for stages where one task takes orders of magnitude longer than the rest—this is your primary indicator of skew.

The Salting Strategy

To balance the workload, we use a technique called salting. Instead of joining on the skewed key alone, you append a random integer (the "salt") to the key in both tables.

Imagine joining Users and Orders on user_id. If user_id = 123 has millions of records:

In the Orders table, transform the key: user_id_salted = concat(user_id, '_', floor(rand() * N)). This spreads the millions of 123 records across N distinct shards.
In the Users table, explode the data: Create N copies of each user row, each with a salt from 0 to N-1.

Now, when you join on user_id_salted, the work is distributed evenly across your cluster nodes because each node only handles a specific salt-shard of the skewed key.

-- Conceptual Salting Join
SELECT *
FROM Orders o
JOIN (
  SELECT *, explode(sequence(0, 19)) as salt FROM Users
) u ON o.user_id = u.user_id 
   AND concat(o.user_id, '_', floor(rand() * 20)) = concat(u.user_id, '_', u.salt)

By artificially increasing the cardinality of your join key, you prevent a single executor from becoming the bottleneck of the entire pipeline.

Key Takeaway: When execution plans reveal a single task dragging down performance, use salting to distribute skewed keys across multiple nodes, ensuring balanced cluster utilization.