In this lesson, you will learn how to extract, flatten, and transform hierarchical data stored within Spark Array and Struct types.
When working with modern data formats like JSON or Parquet, data often arrives in a "nested" state—where a single row contains complex objects or lists rather than simple scalar values. Think of a Struct as a "mini-row" inside a column (e.g., user.address.zipcode) and an Array as a list of multiple values associated with a single record (e.g., user.purchased_items).
Manipulating Structs
Accessing nested fields within a Struct is straightforward; you use dot notation to traverse the path. If you have a column named customer containing sub-fields, you refer to them as customer.name or customer.id. You can treat these fields exactly like top-level columns when using select() or withColumn().
Flattening Arrays
Arrays are more challenging because they represent one-to-many relationships. To manipulate individual elements, Spark provides functions like explode(). This is a powerful transformation that creates a new row for every item in the array, effectively "flattening" the structure.
Consider this example where a customer has an array of tags:
from pyspark.sql.functions import explode
# Suppose 'df' has a column 'tags' which is an array: ["tech", "finance"]
# The explode function turns one row into two rows: one for "tech" and one for "finance"
flattened_df = df.select("customer_id", explode("tags").alias("individual_tag"))
By using explode, you convert a nested list into a format compatible with standard relational operations like groupBy() and filter(). If you only need to inspect or transform elements without creating new rows, you can use functions like transform() to apply logic to every element within the array while keeping the row count intact.
Mastering these functions allows you to bridge the gap between semi-structured data sources and the structured, relational tables required for downstream analytics.
Key Takeaway: Use dot notation to navigate Struct fields and the explode function to unpack Array elements into separate rows for standard relational processing.