Tuning JVM Garbage Collection for Kafka Brokers

Learners will analyze and adjust Java Garbage Collection settings to minimize pause times and maintain stable performance in high-throughput streaming environments.

In this lesson, you will learn how to configure JVM Garbage Collection (GC) settings to minimize "Stop-the-World" pauses that can otherwise destabilize your Kafka brokers.

The Problem with GC Pauses

Kafka is designed for high-throughput, low-latency streaming. However, because Kafka brokers run on the JVM, they are subject to Garbage Collection. When the JVM pauses to reclaim memory, it stops all application threads. If these pauses last too long, the broker may fail to send heartbeats to the ZooKeeper/KRaft controller, leading to unnecessary leader elections, increased replication lag, or even the broker being dropped from the In-Sync Replica (ISR) set.

Choosing the Right Collector

For modern Kafka brokers, the G1 Garbage Collector (G1GC) is the industry standard. It is designed to meet soft real-time latency goals by partitioning the heap into regions and collecting them incrementally.

To tune G1GC, we focus on the MaxGCPauseMillis parameter. This setting acts as a target for the JVM, telling it how long it should attempt to keep each GC pause duration.

Applying the Tuning

You configure these settings in the KAFKA_JVM_PERFORMANCE_OPTS environment variable. A typical production starting point looks like this:

export KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M"

-XX:+UseG1GC: Enables the G1 collector.
-XX:MaxGCPauseMillis=20: Sets a target pause time goal of 20 milliseconds. Be careful not to set this too low, or the JVM will consume excessive CPU trying to meet an impossible goal.
-XX:InitiatingHeapOccupancyPercent=35: Triggers the GC cycle when the heap reaches 35% capacity. This helps prevent the heap from filling up too quickly under heavy producer load.
-XX:G1HeapRegionSize=16M: Configures the region size; for brokers with large heaps (e.g., 6GB+), larger regions can improve GC efficiency.

By tuning these parameters, you ensure that Kafka spends less time cleaning memory and more time moving messages, maintaining the stability of your ISRs and the overall cluster health.

Key Takeaway: Use G1GC with a sensible MaxGCPauseMillis target to prevent long garbage collection pauses that trigger unnecessary leader elections and partition rebalancing.