Course contentsShow
← Data Science
Lesson 1779 of 2,14542. Working with Big DataFree lesson

Reading and Writing Data in Spark

Loading data from various sources (CSV, Parquet, JSON, databases) and writing results efficiently.

Reading and Writing Data in Spark

You'll learn how to load data from multiple sources into Spark and write results back efficiently, handling formats like CSV, Parquet, JSON, and databases.

The Core Idea

Think of Spark as a universal data translator at a massive scale. Just as you need different adapters to connect various devices, Spark provides unified APIs to read from and write to numerous data sources—whether they're simple text files, optimized columnar formats, or remote databases. The beauty is that once data is loaded into a DataFrame, you work with it identically regardless of where it came from.

Reading Data from Various Sources

Spark's DataFrameReader handles input. The general pattern:

df = spark.read.format("source_type").option("key", "value").load("path")

CSV files:

df = spark.read.csv("users.csv", header=True, inferSchema=True)

Parquet (columnar, compressed, faster):

df = spark.read.parquet("data/users.parquet")

JSON:

df = spark.read.json("logs.json")

Databases (JDBC):

df = spark.read.jdbc(url="jdbc:postgresql://host/db", 
                      table="customers", 
                      properties={"user": "admin", "password": "pass"})

Writing Data Efficiently

The DataFrameWriter saves results:

df.write.mode("overwrite").parquet("output/results.parquet")

Modes: overwrite, append, ignore, errorIfExists

Parquet is typically preferred for output—it's compressed, stores schema, and enables fast columnar reads downstream.

Why This Matters

Different formats suit different needs: CSV for human readability, Parquet for performance, JSON for nested data, databases for real-time applications. Spark abstracts these differences, letting you focus on transformations rather than I/O mechanics.

Key Takeaway: Spark provides a unified interface for reading and writing data across diverse sources—master the format options and modes to efficiently move data in and out of your distributed processing pipeline.