Implementing Micro-Batching for Data Lake Ingestion

In this lesson, you will learn how to configure streaming sinks to aggregate events into optimized file formats to solve the "small-file problem" in your data lake.

The Small-File Problem

When ingesting streaming data directly into a data lake (like S3 or HDFS), a common pitfall is writing every incoming event as its own individual file. Because streaming engines trigger writes frequently, you end up with millions of tiny files. This creates massive metadata overhead for your query engines (like Trino or Spark), significantly slowing down read performance and increasing storage costs.

Micro-Batching as a Solution

To mitigate this, we use micro-batching at the sink layer. Instead of flushing every event immediately, we configure the sink to buffer records in memory for a specific duration or until a certain data size threshold is reached. By aggregating these records into a single Parquet or Avro file, we ensure that files are large enough for efficient parallel scanning.

Example Configuration

If you are using a Flink-based streaming pipeline, you typically configure the file sink to use a "rolling policy." This dictates when a file is closed and persisted to the data lake:

FileSink<String> sink = FileSink
    .forBulkFormat(outputPath, ParquetAvroWriters.forReflectRecord(MyEvent.class))
    .withRollingPolicy(OnCheckpointRollingPolicy.build())
    .withBucketAssigner(new DateTimeBucketAssigner<>("yyyy-MM-dd-HH"))
    .build();

In the example above, the OnCheckpointRollingPolicy ensures that files are finalized during each checkpoint. By aligning the file-rolling mechanism with your checkpointing frequency, you gain two benefits:

1. Performance: You control the file size by balancing your checkpoint interval with your throughput.
2. Consistency: Because you are utilizing your distributed snapshotting mechanisms, you ensure that the data written to the lake remains consistent even if a failure occurs.

By moving from "streaming-per-event" to "streaming-per-batch-file," you transform a fragmented data lake into a high-performance analytics layer.

Key Takeaway: Use micro-batching sinks to aggregate events into optimized file formats like Parquet, ensuring that file closing policies are aligned with your checkpointing interval to balance ingestion latency with query performance.