Course contentsShow
← System Design
Lesson 724 of 1,91919. Event Streaming PlatformsFree lesson

Kafka Performance Tuning

Optimizing throughput and latency via batching, compression, acks, buffer sizes, and partition configuration.

Kafka Performance Tuning

What you'll learn: How to optimize Kafka's throughput and latency by tuning batching, compression, acknowledgments, buffers, and partition configuration.

Why Performance Tuning Matters

Kafka can handle millions of messages per second, but only when properly configured. Out-of-the-box settings prioritize safety over speed. Understanding key tuning knobs lets you balance durability, latency, and throughput for your specific use case.

Key Configuration Levers

Batching: Producers don't send every message immediately. Instead, they batch messages together using linger.ms (how long to wait) and batch.size (how many bytes to accumulate). Think of it like waiting for a bus to fill before departing—you trade a tiny delay for much higher throughput.

Compression: Enable compression.type (gzip, snappy, lz4, zstd) to reduce network and disk I/O. Snappy offers good compression with low CPU cost; gzip compresses more but uses more CPU. This is like vacuum-sealing clothes before shipping—same data, smaller package.

Acknowledgments (acks): Controls durability vs speed. acks=0 means fire-and-forget (fastest, no durability), acks=1 waits for leader confirmation (balanced), acks=all waits for all in-sync replicas (slowest, most durable).

Buffer Sizes: Tune buffer.memory (producer-side memory pool) and fetch.min.bytes/fetch.max.wait.ms (consumer pull behavior). Larger buffers smooth out network hiccups but use more memory.

Partition Configuration: More partitions enable higher parallelism and throughput, but increase coordination overhead and rebalancing time. Too few partitions bottleneck throughput; too many add latency.

The Tradeoff Triangle

Every tuning decision involves tradeoffs: throughput vs latency vs durability. Faster batching increases throughput but adds latency. Lower acks speeds writes but risks data loss.

Key Takeaway: Kafka performance tuning requires balancing batching size, compression type, acknowledgment level, buffer allocation, and partition count based on whether your workload prioritizes throughput, latency, or durability.