In this lesson, you will learn how to scale your Kafka Connect tasks to saturate available bandwidth and process high-volume data streams without bottlenecks.
The Anatomy of Parallelism
In Kafka Connect, throughput is determined by the number of tasks assigned to a connector. Unlike consumer groups where one consumer typically handles one partition, a single connector instance can be configured to run multiple tasks in parallel. Each task acts as an independent thread of execution, reading from or writing to your external data systems.
If your source or sink system is high-performance (like an optimized database or a high-IOPS storage bucket), but your connector is lagging, you are likely hitting a parallelism ceiling. You can increase this by tuning the tasks.max configuration.
Tuning for Throughput
Increasing tasks.max allows the Connect worker to spin up more threads, but this is governed by two constraints:
- Partition Bottleneck: A source connector cannot process more tasks than there are partitions in the source Kafka topic. If you have 5 partitions, setting
tasks.maxto 20 won't improve performance because 15 tasks will sit idle. - Resource Contention: Each task consumes CPU and memory. If you increase the count too aggressively, you risk "context switching" overhead or memory pressure, which actually decreases throughput.
To optimize, start by setting tasks.max equal to the number of partitions in your topic. If performance remains sluggish, look at the worker configuration. You can scale horizontally by adding more worker nodes to the Connect cluster; Kafka Connect will automatically rebalance the assigned tasks across these nodes using the same cooperative sticky rebalancing logic you’ve already mastered.
# Example configuration for a high-throughput sink
name=my-optimized-sink
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
tasks.max=10
# Ensure this matches or is less than the source topic partition count
topics=user-events
To verify your configuration, monitor the worker's thread pool utilization. If you see high CPU usage but low throughput, consider optimizing the Single Message Transformations (SMTs) to reduce the computational cost per record before it hits the sink.
Key Takeaway: Maximize connector throughput by aligning tasks.max with your topic partition count and scaling worker nodes horizontally to prevent resource saturation.