Common DataFrame Operations: Select, Filter, GroupBy
You'll learn how to perform essential data manipulation tasks—selecting columns, filtering rows, and aggregating groups—using Spark's distributed DataFrame API.
Core Idea in Plain English
Just like working with pandas DataFrames, Spark DataFrames support the same familiar operations for exploring and transforming data. The key difference? These operations execute across multiple machines in parallel. When you select columns, filter rows, or group data in Spark, the engine automatically distributes the work across your cluster, processing millions or billions of rows efficiently.
The Three Essential Operations
Select: Choosing Columns
select() lets you pick specific columns from your DataFrame, just like SQL's SELECT statement. You're narrowing the dataset horizontally, keeping only the fields you need for analysis.
# Select specific columns
df.select("user_id", "purchase_amount")
Filter: Choosing Rows
filter() (or where()) keeps only rows meeting your condition. This is how you subset data based on criteria—think of it as vertical slicing.
# Keep only high-value purchases
df.filter(df["purchase_amount"] > 100)
GroupBy: Aggregating Data
groupBy() groups rows by one or more columns, then applies aggregate functions like count(), sum(), or avg(). This is your tool for summarization and rollups.
# Total purchases per user
df.groupBy("user_id").sum("purchase_amount")
The Distribution Advantage
Behind the scenes, Spark partitions your data across worker nodes. When you chain these operations, Spark optimizes the execution plan and processes each partition independently—no single bottleneck.
Key Takeaway: Spark DataFrames support select, filter, and groupBy operations with familiar syntax, but execute them in parallel across distributed data partitions for massive scalability.