Articles in this section

See more

Resolving Kafka Pipeline Batch Failures: When and How to Use the operation.timeout.ms Parameter

{question}

How can we resolve Kafka pipeline batch failures?

{question}

{answer}

Overview

The operation.timeout.ms parameter controls how long the Kafka client operations (such as metadata fetches and produce/consume requests) wait before timing out (default value: 10s). Adjusting this parameter can help address issues where pipelines fail due to network delays or slow broker responses, especially after upgrading SingleStoreDB or when ingesting from remote or secured Kafka clusters.


When to Add operation.timeout.ms

  • Timeout Errors During Pipeline Batches: If you see errors of the following type in your pipeline logs, this indicates that Kafka operations are not finishing within the default timeout:
Batch starting with new consumer.  rd_kafka_version_str: 1.9.2-21-gff6b70
2024-05-30 07:25:42.955 Timed out streaming message with error Local: Timed out
2024-05-30 07:25:52.965 Timed out streaming message with error Local: Timed out
2024-05-30 07:26:02.976 Timed out streaming message with error Local: Timed out
2024-05-30 07:26:02.986 _$_SERVER_ERROR

  • Network Latency or Broker Connectivity Issues: When your Kafka brokers are hosted remotely, behind SSL, or if there are intermittent network issues, a higher timeout allows more time for connections and metadata retrieval to be successful.

How to Add or Adjust operation.timeout.ms to an existing pipeline:

Add or increase the operation.timeout.ms parameter in your pipeline’s Kafka configuration. For example:

 
CREATE OR REPLACE PIPELINE p AS LOAD DATA kafka 'host.example.com:9092/whatever2' 
CONFIG '{
"operation.timeout.ms": "60000" 
}'
  INTO TABLE t format CSV;

This sets the timeout to 60 seconds. Adjust the value according to your network conditions and the error patterns you observe.

Recommended Circumstances:

  • Persistent Local: Timed out or Local: Broker transport failure errors in pipeline logs.

  • Pipelines that intermittently fail to fetch Kafka metadata or connect to brokers.

  • When all other connectivity and configuration checks (network, SSL certificates, broker availability) have been verified, and issues persist

Summary Table:

Symptom Action
Timed out streaming/fetching Kafka messages Add/increase operation.timeout.ms
Connection setup timed out in state CONNECT Add/increase operation.timeout.ms
Broker transport failures Add/increase operation.timeout.ms

Note: Always ensure that your Kafka brokers are reachable and that network or firewall settings permit uninterrupted connectivity before increasing timeouts.

{answer}

Related articles