Reading and Writing Data in Spark
You'll learn how to load data from multiple sources into Spark and write results back efficiently, handling formats like CSV, Parquet, JSON, and databases.
The Core Idea
Think of Spark as a universal data translator at a massive scale. Just as you need different adapters to connect various devices, Spark provides unified APIs to read from and write to numerous data sources—whether they're simple text files, optimized columnar formats, or remote databases. The beauty is that once data is loaded into a DataFrame, you work with it identically regardless of where it came from.
Reading Data from Various Sources
Spark's DataFrameReader handles input. The general pattern:
df = spark.read.format("source_type").option("key", "value").load("path")
CSV files:
df = spark.read.csv("users.csv", header=True, inferSchema=True)
Parquet (columnar, compressed, faster):
df = spark.read.parquet("data/users.parquet")
JSON:
df = spark.read.json("logs.json")
Databases (JDBC):
df = spark.read.jdbc(url="jdbc:postgresql://host/db",
table="customers",
properties={"user": "admin", "password": "pass"})
Writing Data Efficiently
The DataFrameWriter saves results:
df.write.mode("overwrite").parquet("output/results.parquet")
Modes: overwrite, append, ignore, errorIfExists
Parquet is typically preferred for output—it's compressed, stores schema, and enables fast columnar reads downstream.
Why This Matters
Different formats suit different needs: CSV for human readability, Parquet for performance, JSON for nested data, databases for real-time applications. Spark abstracts these differences, letting you focus on transformations rather than I/O mechanics.
Key Takeaway: Spark provides a unified interface for reading and writing data across diverse sources—master the format options and modes to efficiently move data in and out of your distributed processing pipeline.