BiteSizedChunks.comLearn one small thing at a time.
Course contentsShow
← Data Engineering
Lesson 535 of 8258. Real-time Data Streaming with KafkaFree lesson

Achieving Durability with Acks Configuration

Configure producer acknowledgement settings to ensure data is successfully written to the required number of replicas before confirming ingestion.

In this lesson, you will learn how to calibrate the acks configuration in Kafka producers to balance the trade-off between system performance and data durability.

When a producer sends a record to a Kafka partition, it needs to know whether the broker has safely stored the message. The acks setting dictates exactly how many replicas must acknowledge the write before the broker sends a success response back to the producer.

The Three Levels of Acknowledgment

You can set the acks parameter to one of three values:

  1. acks=0: The producer does not wait for any acknowledgment from the broker. This provides the highest throughput because the producer doesn't wait for a response, but it carries the highest risk of data loss if the broker crashes immediately after receiving the record.
  2. acks=1: The producer waits for the leader replica to write the record to its local log. This provides a basic level of durability; if the leader fails before the followers replicate the data, the message is lost.
  3. acks=all (or -1): The producer waits for the full set of In-Sync Replicas (ISR) to acknowledge the write. This is the gold standard for durability. As long as at least one in-sync replica remains alive, your data is guaranteed to persist even if the leader fails.

Implementation Example

To ensure your producer is configured for maximum durability, you define the setting in your producer properties:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
// Ensure data is written to all ISR before confirming success
props.put("acks", "all"); 
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

By choosing acks=all, you are instructing Kafka to honor the durability guarantees defined by your ISR settings. If you have configured your cluster with min.insync.replicas, the broker will ensure that the required number of replicas have confirmed the write before the producer receives an acknowledgment.

Key Takeaway: Configuring acks=all provides the strongest durability guarantee by requiring that all in-sync replicas successfully persist the record before the producer considers the write complete.