In this lesson, you will learn how to identify the performance bottlenecks caused by excessive small files and understand why they degrade your read latency and metadata efficiency.
The Small File Problem
In a distributed data lake (like Delta Lake or Iceberg), data is stored as individual files in object storage (like S3 or ADLS). While the system is designed to handle massive datasets, it struggles when data is broken into millions of tiny files (e.g., a few kilobytes or megabytes each). This is often called the "Small File Problem."
Impact on Metadata Overhead
Every time you query a table, the engine must open the metadata layer (like the transaction log or manifest files) to identify which files to read. If you have 100,000 tiny files, the query engine must perform an expensive "list" operation on your storage and process a massive metadata manifest before it even begins reading the actual data. This "metadata tax" creates a significant bottleneck before your query execution even starts.
Impact on Read Latency
Distributed systems are optimized for "bulk" reads. Opening a file incurs a fixed overhead—the time it takes to authenticate, establish a connection, and seek the start of the data. If your data is spread across thousands of small files, the query engine spends more time opening and closing files than actually processing your data.
Consider this visualization of a query plan:
-- Scenario A: 1 large file (1GB)
-- Read overhead: Low (1 connection)
-- Scenario B: 1,000 small files (1MB each)
-- Read overhead: Extremely High (1,000 connections/opens)
How to Diagnose
You can diagnose this by checking the file count against your data size. If your table size is 10GB but consists of 50,000 files, your average file size is only 0.2MB. This is a clear indicator that your read performance is being throttled by metadata and I/O overhead. You should look for symptoms like:
- High "Planning Time" in your query logs (time spent before execution starts).
- High "Total Time" for simple filters that should be fast but are sluggish.
Key Takeaway: Excessive small files turn metadata lookups into a massive bottleneck and force the query engine to spend more time establishing connections than actually reading your data.