In this lesson, you will learn how to reorganize fragmented, small files into larger, optimized blocks to drastically accelerate query performance and throughput.
The Small File Problem
When streaming data into a data lake or performing frequent, incremental micro-batches, your storage layer often ends up cluttered with thousands of tiny files. While this keeps latency low for individual writes, it creates a "metadata explosion." When a query engine like Spark or Trino tries to read this data, it must perform thousands of separate I/O operations to open, scan, and close these small files. This overhead often dwarfs the actual time spent processing data, leading to sluggish performance.
The Compaction Strategy
Compaction is the process of periodically merging these fragmented files into larger, uniform blocks (typically 128MB to 1GB in size). By doing this, we reduce the total number of files the query engine must track and open, allowing for more efficient parallel scans and optimized data skipping.
Think of it like cleaning your desk: instead of having 500 loose sticky notes scattered everywhere, you organize the information into a few clean, consolidated folders.
Implementing Basic Compaction
The logic involves reading the fragmented small files and writing them back out into a smaller number of files. In a modern Lakehouse architecture, you might execute a command like this:
-- Consolidating small files into 256MB blocks
OPTIMIZE table_name
WHERE partition_date = '2023-10-01'
WITH (target_file_size_bytes = 268435456);
Behind the scenes, the compute engine reads the small files, re-partitions the data in memory, and writes new, optimally sized Parquet files. Once the write succeeds, the metadata layer updates to point to these new, larger blocks, and the old, fragmented files are marked for deletion (garbage collection).
By controlling the file size, you ensure the storage layout matches the way your distributed compute engine prefers to read data—large, sequential reads rather than a "random seek" nightmare. This reduces metadata overhead and ensures your analytical queries spend their time processing data, not negotiating with the file system.
Key Takeaway: Compaction transforms high-frequency, write-optimized small files into read-optimized, large Parquet blocks, ensuring efficient I/O and faster query execution.