Kafka Connect Sink Backpressure Isolation in High-Throughput Streams

Kafka Connect Sink Backpressure Isolation broke down during a 400% surge in database CDC payloads, driving consumer lag to 1.8 million messages and triggering cascading task restarts across the distributed cluster. When downstream relational databases or search indexes stall on heavy lock contention or I/O bottlenecks, sink connectors unthinkingly pull raw data from brokers until JVM heap limits are breached. The default behavior of Kafka Connect worker tasks—polling batches as fast as the network layer allows while relying on unbounded in-memory buffers—transforms localized storage slowness into cluster-wide outages. Understanding how to bound memory allocation, tune consumer polling rates, and build explicit backpressure triggers inside sink task execution loops is mandatory for maintaining sub-second freshness SLOs.

Root Cause Analysis: JVM Heap Exhaustion and Consumer Group Rebalance Storms

When a downstream destination like PostgreSQL, Elasticsearch, or Snowflake slows down, a Kafka Connect sink worker enters a dangerous execution state. The framework uses internal poll threads that retrieve records from broker partitions via KafkaConsumer.poll() and hand them over to SinkTask.put(). If the destination database slows from writing 10,000 records per second to 200 records per second due to transaction log lock contention, the sink task spend significantly more time inside the put() invocation or in subsequent batch flush cycles.

While SinkTask.put() is blocked waiting for database TCP sockets or connection pool checkouts, the underlying consumer poll loop pauses. If this pause exceeds the configured max.poll.interval.ms (defaulting to 300,000 milliseconds or 5 minutes), the Kafka broker marks the consumer thread dead. The group coordinator triggers a consumer group rebalance across all workers in the cluster. During the rebalance, partition assignments are revoked, in-flight transaction batches are rolled back, and uncommitted offsets cause downstream duplications. Once the rebalance completes, the newly assigned workers immediately attempt to consume the exact same uncommitted large batch, recreating the write slowdown and triggering another rebalance storm.

Simultaneously, memory pressure spikes inside the JVM heap. If the connector buffers uncommitted records in custom internal lists before executing bulk insert operations, the garbage collector struggles to reclaim short-lived objects. Stop-the-world (STW) GC pauses worsen the delay, making it impossible for worker nodes to respond to heartbeat requests on the heartbeat.interval.ms thread. As a result, nodes are evicted from the cluster, causing cascading failures across unrelated connector tasks running on the same shared cluster node.

Bounding Memory Footprints with Consumer Group Configuration

Eliminating memory exhaustion requires tight bounds on worker consumption mechanics before records ever enter the application layer. The primary lever for adjusting this pipeline flow is constraining max.poll.records and sizing Java heap limits relative to maximum payload dimensions.