Loading Structured Datasets into Spark DataFrames

Learn to ingest data from various file formats like Parquet, JSON, and CSV into immutable distributed DataFrames.

In this lesson, you will learn how to ingest diverse file formats—Parquet, JSON, and CSV—into immutable distributed DataFrames to serve as the foundation for your Spark transformations.

Understanding Data Ingestion

When working with Spark, the entry point for your data processing pipeline is the DataFrameReader. Because Spark operates on a distributed architecture, it doesn’t "load" a file into memory on a single machine; instead, it uses your specified file path to create a logical plan that tells the executors where to read the data across the cluster once an action is triggered.

Loading Different Formats

Spark provides a unified interface to read data. The most important aspect is ensuring your data is structured correctly so that Spark can infer the schema or use a provided one.

Parquet: As a columnar storage format, Parquet is the native "best friend" of Spark. It stores schema metadata within the file, making ingestion extremely efficient.
JSON: Since JSON is semi-structured, Spark performs a schema-inference pass. While convenient, this requires an extra scan of the data, which can be costly for massive datasets.
CSV: CSVs require explicit configuration (like defining delimiters or headers) to be parsed correctly into a relational structure.

Here is how these look in practice:

# Loading a Parquet file (Spark's preferred format)
df_parquet = spark.read.format("parquet").load("s3://data/events.parquet")

# Loading JSON with schema inference
df_json = spark.read.option("inferSchema", "true").json("s3://data/logs.json")

# Loading CSV with header parsing
df_csv = spark.read.option("header", "true").csv("s3://data/users.csv")

The Immutable Nature

Once these files are loaded into a DataFrame, the object is immutable. You cannot change the contents of the DataFrame directly. Any transformation you apply will produce a new DataFrame. This immutability is what allows Spark to maintain the integrity of your Directed Acyclic Graph (DAG) and ensures fault tolerance—if a partition is lost, Spark can simply re-read the source file and re-apply the transformation.

Key Takeaway: Loading data into Spark is a metadata-driven operation that establishes the source for your distributed execution plan, leveraging file-specific readers to convert raw bytes into structured, immutable DataFrames.